CDH搭建大数据集群（5.10.0）

[client]
port            = 3306
socket          = /usr/local/mysql/data/mysql.sock
 
[mysqld]
port            = 3306
socket          = /usr/local/mysql/data/mysql.sock

skip-external-locking
key_buffer_size = 256M
sort_buffer_size = 2M
read_buffer_size = 2M
read_rnd_buffer_size = 4M
query_cache_size= 32M
max_allowed_packet = 16M
myisam_sort_buffer_size=128M
tmp_table_size=32M

table_open_cache = 512
thread_cache_size = 8
wait_timeout = 86400
interactive_timeout = 86400
max_connections = 600

thread_concurrency = 32


default-storage-engine = INNODB
transaction-isolation = READ-COMMITTED

server-id  = 1
basedir     = /usr/local/mysql
datadir     = /usr/local/mysql/data
pid-file     = /usr/local/mysql/data/hostname.pid


log-warnings
sysdate-is-now

binlog_format = MIXED
log_bin_trust_function_creators=1
log-error  = /usr/local/mysql/data/hostname.err
log-bin=/usr/local/mysql/arch/mysql-bin

innodb_data_home_dir = /usr/local/mysql/data/
innodb_data_file_path = ibdata1:500M:autoextend
innodb_log_group_home_dir = /usr/local/mysql/arch
innodb_log_files_in_group = 2
innodb_log_file_size = 200M


innodb_buffer_pool_size = 1024M
innodb_additional_mem_pool_size = 50M
innodb_log_buffer_size = 16M

innodb_lock_wait_timeout = 100
innodb_flush_log_at_trx_commit = 1
innodb_locks_unsafe_for_binlog=1

performance_schema
innodb_read_io_threads=4　　
innodb-write-io-threads=4
innodb-io-capacity=200
innodb_purge_threads=1
innodb_use_native_aio=on

innodb_file_per_table = 1
lower_case_table_names=1

[mysqldump]
quick
max_allowed_packet = 16M

[mysql]
no-auto-rehash

[mysqlhotcopy]
interactive-timeout

[myisamchk]
key_buffer_size = 256M
sort_buffer_size = 256M
read_buffer = 2M
write_buffer = 2M

5、修改my.cnf文件的属性和权限

依次执行以下命令：

[root@hadoop01 etc]# chown mysqladmin:dba /etc/my.cnf
[root@hadoop01 etc]# chmod 640 /etc/my.cnf
[root@hadoop01 etc]# chown -R mysqladmin:dba /usr/local/mysql
[root@hadoop01 etc]# chmod -R 755 /usr/local/mysql
[root@hadoop01 etc]# su - mysqladmin
[mysqladmin@hadoop01 ~]??? /???/?????/????? [??????????@ℎ?????01 ]pwd /usr/local/mysql [mysqladmin@hadoop01 ] mkdir arch backup

执行初始化脚本，打印的日志没有报错，说明运行ok：
[mysqladmin@hadoop01 ~]$

初始化脚本命令：scripts/mysql_install_db --user=mysqladmin --basedir=/usr/local/mysql --datadir=/usr/local/mysql/data

6、配置mysql服务和自启动

在root用户下执行：

[mysqladmin@hadoop01 ~]$ su root
Password:
[root@hadoop01 mysql]#
[root@hadoop01 mysql]# cd /usr/local/mysql
[root@hadoop01 mysql]# cp /usr/local/mysql/support-files/mysql.server /etc/rc.d/init.d/mysql
[root@hadoop01 mysql]# chmod +x /etc/rc.d/init.d/mysql
[root@hadoop01 mysql]# chkconfig --del mysql
[root@hadoop01 mysql]# chkconfig --add mysql
[root@hadoop01 mysql]# chkconfig --level 345 mysql on
[root@hadoop01 mysql]# vim /etc/rc.local（将里面的内容都删掉，拷贝以下内容）

#!/bin/sh
#
# This script will be executed *after* all the other init scripts.
# You can put your own initialization stuff in here if you don't
# want to do the full Sys V style init stuff.

touch /var/lock/subsys/local

su - mysqladmin -c "/etc/init.d/mysql start --federated"

7、启动mysql并监听进程

执行以下命令：

[root@hadoop01 mysql]# su - mysqladmin
[mysqladmin@hadoop01 ~]$ mysqld_safe &
[1] 1888

重新打开一个连接执行：

ps -ef|grep mysqld（查看mysql的进程是否运行）

service mysql status（查看mysql的运行状态）

出现上图代表启动ok

8、修改mysql的密码

执行以下命令：

mysql> use mysql

mysql> update user set password=password('root') where user='root';

mysql> select host,user,password from user;

mysql> delete from user where user='';

mysql> flush privileges;

9、更改.bash_profile文件

进入到mysql目录中，执行vim ./.bash_profile，拷贝以下内容：

[root@hadoop01 mysql]# cd /usr/local/mysql/

[root@hadoop01 mysql]# vim .bash_profile

# .bash_profile
# Get the aliases and functions

if [ -f ~/.bashrc ]; then
        . ~/.bashrc
fi

# User specific environment and startup programs
MYSQL_BASE=/usr/local/mysql
export MYSQL_BASE
PATH=${MYSQL_BASE}/bin:$PATH
export PATH

unset USERNAME

#stty erase ^H
set umask to 022
umask 022
PS1=`uname -n`":"'$USER'":"'$PWD'":>"; export PS1

七.安装http和启动http服务

1.安装http服务

切换到root用户：

[root@hadoop01 mysql]# rpm -qa|grep httpd
[root@hadoop01 mysql]# yum install -y httpd

[root@hadoop01 mysql]# chkconfig --list|grep httpd

日志显示：httpd 0:off 1:off 2:off 3:off 4:off 5:off 6:off

[root@hadoop01 mysql]# chkconfig httpd on

[root@hadoop01 mysql]# chkconfig --list|grep httpd

日志显示：httpd 0:off 1:off 2:on 3:on 4:on 5:on 6:off

[root@hadoop01 mysql]# service httpd start

2.创建parcels文件

执行以下命令：

[root@hadoop01 mysql]# cd /var/www/html

[root@hadoop01 html]# mkdir parcels

将开始下载的三个文件上传至此文件夹下：

①http://archive.cloudera.com/cdh5/parcels/5.10.0/CDH-5.10.0-1.cdh5.10.0.p0.41-el6.parcel

②http://archive.cloudera.com/cdh5/parcels/5.10.0/CDH-5.10.0-1.cdh5.10.0.p0.41-el6.parcel.sha1

③http://archive.cloudera.com/cdh5/parcels/5.10.0/manifest.json

[root@hadoop01 parcels]# mkdir /opt/rpminstall

[root@hadoop01 parcels]# cd /opt/rpminstall

将下载的tarball上传：cm5.10.0-centos6.tar.gz至当前目录下

解压：[root@hadoop01 rpminstall]# tar -xzvf cm5.10.0-centos6.tar.gz -C /var/www/html/

[root@hadoop01 rpminstall]# cd /var/www/html
[root@hadoop01 html]# ll

创建和官网相同的目录：

[root@hadoop01 html]# mkdir -p cm5/redhat/6/x86_64/
[root@hadoop01 html]# mv cm cm5/redhat/6/x86_64/

3.配置本地yum源（3个节点）

[root@hadoop01 ~]# vi /etc/yum.repos.d/cloudera-manager.repo

粘贴以下内容：ip地址为当前机器的ip地址，如果集群在内网中则配置内网ip即可，该文件每台服务器都要配置一个，保存退出！

[cloudera-manager]
name = Cloudera Manager, Version 5.10.0
baseurl = http://39.100.73.64/cm5/redhat/6/x86_64/cm/5/
gpgcheck = 0

浏览器查看下面两个网址是否出来,假如有，就配置成功（以下ip为公网ip）

http://39.100.73.64/parcels/

http://39.100.73.64/cm5/redhat/6/x86_64/cm/5/

回到顶部

八.安装并启动CM服务

1、执行以下命令

[root@hadoop01 ~]# cd /var/www/html/cm5/redhat/6/x86_64/cm/5/RPMS/x86_64

[root@hadoop01 x86_64]# yum install -y cloudera-manager-daemons-5.10.0-1.cm5100.p0.85.el6.x86_64.rpm

[root@hadoop01 x86_64]# yum install -y cloudera-manager-server-5.10.0-1.cm5100.p0.85.el6.x86_64.rpm

顺序不能错，只装这两个

[root@hadoop01 x86_64]# mkdir /usr/share/java

[root@hadoop01 x86_64]# cd /usr/share/java/

将mysql-connector-java.jar上传到该目录下：.jar包名称必须为mysql-connector-java.jar

2、进入到mysql中，创建元数据

执行以下命令：

[root@hadoop01 java]# su - mysqladmin

hadoop01:mysqladmin:/usr/local/mysql:>cd bin

3、进入数据库后，执行以下命令：

mysql> create database cmf DEFAULT CHARACTER SET utf8;

mysql> grant all on cmf.* TO 'cmf'@'%' IDENTIFIED BY 'root';

mysql> create database amon DEFAULT CHARACTER SET utf8;

mysql> grant all on amon.* TO 'amon'@'%' IDENTIFIED BY 'root';

mysql> grant all privileges on *.* to 'root'@'%' identified by 'root' with grant option;

mysql> flush privileges;

切换到root用户：

[root@hadoop01 ~]# cd /etc/cloudera-scm-server/

[root@hadoop01 cloudera-scm-server]# vi db.properties（按照下图中进行配置）

标注内容从上往下分别代表：数据库类型，数据库所在的主机ip:端口，数据库名称，数据库用户，数据库设置类型

标注要和你之前的配置匹配

配置好后，保存退出

4、启动CM服务

执行以下命令：

[root@hadoop01 jdk1.8.0_181]# service cloudera-scm-server start

查看日志：

[root@hadoop01 jdk1.8.0_181]# cd /var/log/cloudera-scm-server/
[root@hadoop01 cloudera-scm-server]# tail -f cloudera-scm-server.log

没有错误日志提示，说明启动成功~~~~~

九.CDH配置

1、登录CDH配置界面

http://39.100.73.64:7180（ip为公网ip）,用户名和密码都是admin，下面正式开始我们的页面配置过程。（需要进入阿里云控制台，将公网ip的端口开放：7180）

2、选择免费

3、配置CDH集群

4、点击搜索

出现这个页面，代表集群能连接上，当前受管这一栏全部为否，如果有是的话，代表之前已经安装好并且没有卸载干净，需要卸载干净后重启服务后在进入到该页面。

5、配置parcels文件

5.1、点击更多选项

5.2、配置远程 Parcel 存储库 URL

进入到该页面中，远程 Parcel 存储库 URL这一栏删掉只留下一个，将内容更改为之前配置过的parcel地址，这里用的是内网的ip，所以是http://39.100.73.64/parcels/，点击保存

5.3、选择版本和自定义存储库

5.4、不勾选JDK

5.5、不勾选单用户模式

5.6、设置主机密码

5.7、gent客户端安装

等待agent客户端安装，这一步可能会出现各种问题，通过点击出现问题的服务器的详细信息查看出现问题的地方并更改之后重启服务重新安装

6.等待安装完成后，点击继续

7.等待安装分配完成后，点击继续

8.继续等待检查主机

9.这里出现了警告,下面解决警告

透明大页面和swap值需要更改

将每台机器关闭大页面

执行以下命令：

在每个节点执行以下命令：

[root@hadoop01 cloudera-scm-server]# echo never > /sys/kernel/mm/transparent_hugepage/defrag
[root@hadoop01 cloudera-scm-server]# echo never > /sys/kernel/mm/transparent_hugepage/enabled
[root@hadoop01 cloudera-scm-server]# echo 'echo never > /sys/kernel/mm/transparent_hugepage/defrag'>> /etc/rc.local
[root@hadoop01 cloudera-scm-server]# echo 'echo never > /sys/kernel/mm/transparent_hugepage/enabled'>> /etc/rc.local

[root@hadoop01 cloudera-scm-server]# echo 'vm.swappiness = 10' >> /etc/sysctl.conf

[root@hadoop01 cloudera-scm-server]# sysctl –p