安装前准备
环境：
CDH 6.3
CentOS 7
1.修改主机名，配置hosts文件，将集群配置到hosts文件中
2.机器关闭防火墙
[root@cdh01 ~]# systemctl stop firewalld && systemctl disable firewalld && systemctl status firewalld
● firewalld.service - firewalld - dynamic firewall daemon
Loaded: loaded (/usr/lib/systemd/system/firewalld.service; disabled; vendor preset: enabled)
Active: inactive (dead)
Docs: man:firewalld(1)
3.机器之间root用户配置ssh免密登录
[root@cdh01 ~]# ssh-keygen -t rsa
[root@cdh01 ~]# cat ~/.ssh/id_rsa.pub >> authorized_keys
[root@cdh01 ~]# ssh localhost
Permission denied (publickey,gssapi-keyex,gssapi-with-mic).
[root@cdh01 ~]# vim /etc/ssh/sshd_config
PermitRootLogin yes
[root@cdh01 ~]# systemctl restart sshd
[root@cdh01 ~]# ssh localhost
Last login: Wed Oct 9 10:04:38 2019
所有机器执行以上操作后，将主节点的公钥复制到所有子节点的authorized_keys文件中即可。当然为了集群跳转方便，也可以将集群所有机器的公钥都复制一份。
4.机器之间配置时间同步
[root@cdh01 :~ ]# yum install ntp
[root@cdh01 :~ ]# vim /etc/ntp.conf
driftfile /var/lib/ntp/drift
restrict default kod nomodify notrap nopeer noquery
restrict -6 default kod nomodify notrap nopeer noquery
restrict 127.0.0.1
restrict -6 ::1
server localhost1 ###ntp时间同步地址
server localhost2 ###ntp时间同步地址
includefile /etc/ntp/crypto/pw
keys /etc/ntp/keys
[root@cdh01 :~ ]# systemctl start ntpd
[root@cdh01 :~ ]# systemctl status ntpd
[root@cdh01 :~ ]# systemctl enable ntpd
5.在cloudera manager server节点上安装MySQL
MySQL下载地址，下载什么版本自己根据情况决定，我这里下载的是mysql-5.7.27-1.el7.x86_64.rpm-bundle.tar，里面包含所有安装MySQL需要的RPM包。
解压MySQL包
[root@cdh01 ~]# tar -xvf mysql-5.7.27-1.el7.x86_64.rpm-bundle.tar -C .
mysql-community-libs-5.7.27-1.el7.x86_64.rpm
mysql-community-embedded-devel-5.7.27-1.el7.x86_64.rpm
mysql-community-libs-compat-5.7.27-1.el7.x86_64.rpm
mysql-community-devel-5.7.27-1.el7.x86_64.rpm
mysql-community-embedded-compat-5.7.27-1.el7.x86_64.rpm
mysql-community-common-5.7.27-1.el7.x86_64.rpm
mysql-community-client-5.7.27-1.el7.x86_64.rpm
mysql-community-server-5.7.27-1.el7.x86_64.rpm
mysql-community-test-5.7.27-1.el7.x86_64.rpm
mysql-community-embedded-5.7.27-1.el7.x86_64.rpm
先删除服务器上相关的依赖，否则安装会冲突
[root@cdh01 ~]# rpm -qa |grep mysql
[root@cdh01 ~]# rpm -qa |grep MySQL
[root@cdh01 ~]# rpm -qa |grep mariadb
mariadb-libs-5.5.60-1.el7_5.x86_64
[root@cdh01 ~]# rpm -e mariadb-libs-5.5.60-1.el7_5.x86_64 --nodeps
[root@cdh01 ~]# rpm -qa |grep mariadb
安装MySQL服务
[root@cdh01 ~]# rpm -i mysql-community-common-5.7.27-1.el7.x86_64.rpm
warning: mysql-community-common-5.7.27-1.el7.x86_64.rpm: Header V3 DSA/SHA1 Signature, key ID 5072e1f5: NOKEY
[root@cdh01 ~]# rpm -i mysql-community-libs-5.7.27-1.el7.x86_64.rpm
warning: mysql-community-libs-5.7.27-1.el7.x86_64.rpm: Header V3 DSA/SHA1 Signature, key ID 5072e1f5: NOKEY
[root@cdh01 ~]# rpm -i mysql-community-client-5.7.27-1.el7.x86_64.rpm
warning: mysql-community-client-5.7.27-1.el7.x86_64.rpm: Header V3 DSA/SHA1 Signature, key ID 5072e1f5: NOKEY
[root@cdh01 ~]# rpm -i mysql-community-server-5.7.27-1.el7.x86_64.rpm
warning: mysql-community-server-5.7.27-1.el7.x86_64.rpm: Header V3 DSA/SHA1 Signature, key ID 5072e1f5: NOKEY
查看MySQL服务状态
[root@cdh01 ~]# systemctl status mysqld
● mysqld.service - MySQL Server
Loaded: loaded (/usr/lib/systemd/system/mysqld.service; enabled; vendor preset: disabled)
Active: inactive (dead)
Docs: man:mysqld(8)
http://dev.mysql.com/doc/refman/en/using-systemd.html
启动MySQL服务
[root@cdh01 ~]# systemctl start mysqld
[root@cdh01 ~]# systemctl status mysqld
● mysqld.service - MySQL Server
Loaded: loaded (/usr/lib/systemd/system/mysqld.service; enabled; vendor preset: disabled)
Active: active (running) since Wed 2019-10-09 15:37:55 CST; 2s ago
Docs: man:mysqld(8)
http://dev.mysql.com/doc/refman/en/using-systemd.html
Process: 5723 ExecStart=/usr/sbin/mysqld --daemonize --pid-file=/var/run/mysqld/mysqld.pid $MYSQLD_OPTS (code=exited, status=0/SUCCESS)
Process: 5608 ExecStartPre=/usr/bin/mysqld_pre_systemd (code=exited, status=0/SUCCESS)
Main PID: 5726 (mysqld)
CGroup: /system.slice/mysqld.service
└─5726 /usr/sbin/mysqld --daemonize --pid-file=/var/run/mysqld/mysqld.pid
查看MySQL启动后生成的临时密码：
[root@cdh01 ~]# cat /var/log/mysqld.log | grep 'password'
2019-10-09T07:37:51.329693Z 1 [Note] A temporary password is generated for root@localhost: tVq,t)UWZ3a/
登录MySQL修改密码
[root@cdh01 ~]# mysql -uroot -p
Enter password:
Welcome to the MySQL monitor. Commands end with ; or \g.
Your MySQL connection id is 2
Server version: 5.7.27
Copyright (c) 2000, 2019, Oracle and/or its affiliates. All rights reserved.
Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.
Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
mysql>ALTER USER 'root'@'localhost' IDENTIFIED BY 'Password';
6.准备安装所需的CDH软件
CDH
parcels
yum源
7.安装http服务器和createrepo
[root@cdh01 ~]# yum install httpd createrepo
[root@cdh01 ~]# systemctl start httpd
[root@cdh01 ~]# systemctl status httpd
● httpd.service - The Apache HTTP Server
Loaded: loaded (/usr/lib/systemd/system/httpd.service; disabled; vendor preset: disabled)
Active: active (running) since Wed 2019-10-09 11:20:37 CST; 23h ago
Docs: man:httpd(8)
man:apachectl(8)
Main PID: 28016 (httpd)
8.安装MySQL的jdbc驱动
下载MySQL的jdbc驱动，将驱动改名为mysql-connector-java.jar，然后将驱动复制到/usr/share/java目录下(没有该目录则自己创建，所有机器上都要装驱动)。
9.安装jdk
在CDH server节点上安装从CDH下载的oracle-j2sdk1.8-1.8.0+update181-1.x86_64.rpm
10.修改Linux swappiness参数(所有节点）
为了避免服务器使用swap功能而影响服务器性能，一般都会把vm.swappiness修改为0（cloudera建议10以下）
编辑/etc/sysctl.conf文件，在最后添加vm.swappiness=0
上述方法rhel6有效，rhel7.2中:tuned服务会动态调整系统参数
查找tuned中配置，直接修改配置
[root@cdh01 ~]# cd /usr/lib/tuned/
[root@cdh01 tuned]# grep "vm.swappiness" * -R
latency-performance/tuned.conf:vm.swappiness=10
throughput-performance/tuned.conf:vm.swappiness=10
virtual-guest/tuned.conf:vm.swappiness = 30
然后将文件中的配置依次修改为0
修改后将这些文件同步到其他机器上
11.禁用透明页(所有节点）
[root@cdh01 ~]# vim /etc/rc.local
在文件中添加如下内容：
echo never > /sys/kernel/mm/transparent_hugepage/defrag
echo never > /sys/kernel/mm/transparent_hugepage/enabled
然后将该文件同步其他机器上，然后启动所有服务器
12.配置本地Parcel存储库
创建/opt/cloudera/parcel-repo目录，将下载好的parcel和manifest.json文件放置到该目录，然后执行下面的命令：
[root@cdh01 parcel-repo]# sha1sum CDH-6.3.0-1.cdh6.3.0.p0.1279813-el7.parcel | awk '{ print $1 }' > CDH-6.3.0-1.cdh6.3.0.p0.1279813-el7.parcel.sha
[root@cdh01 parcel-repo]# ll
total 2036856
-rwxr-xr-x 1 root root 2085690155 Oct 9 15:29 CDH-6.3.0-1.cdh6.3.0.p0.1279813-el7.parcel
-rw-r--r-- 1 root root 41 Oct 11 14:36 CDH-6.3.0-1.cdh6.3.0.p0.1279813-el7.parcel.sha
-rwxr-xr-x 1 root root 64 Oct 9 14:18 CDH-6.3.0-1.cdh6.3.0.p0.1279813-el7.parcel.sha256
-rwxr-xr-x 1 root root 33887 Oct 11 14:35 manifest.json
安装CDH
创建cloudera-repo目录
[root@cdh01 html]# mkdir cloudera-repo
将下载的下列软件放置到cloudera-repo目录，
allkeys.asc
cloudera-manager-agent-6.3.0-1281944.el7.x86_64.rpm
cloudera-manager-daemons-6.3.0-1281944.el7.x86_64.rpm
cloudera-manager-server-6.3.0-1281944.el7.x86_64.rpm
cloudera-manager-server-db-2-6.3.0-1281944.el7.x86_64.rpm
manifest.json
oracle-j2sdk1.8-1.8.0+update181-1.x86_64.rpm
RPM-GPG-KEY-cloudera
执行createrepo命令，生成repodata目录
[root@cdh01 cloudera-repo]# createrepo .
Spawning worker 0 with 1 pkgs
Spawning worker 1 with 1 pkgs
Spawning worker 2 with 1 pkgs
Spawning worker 3 with 1 pkgs
Spawning worker 4 with 1 pkgs
Spawning worker 5 with 0 pkgs
Spawning worker 6 with 0 pkgs
Spawning worker 7 with 0 pkgs
Spawning worker 8 with 0 pkgs
Spawning worker 9 with 0 pkgs
Spawning worker 10 with 0 pkgs
Spawning worker 11 with 0 pkgs
Spawning worker 12 with 0 pkgs
Spawning worker 13 with 0 pkgs
Spawning worker 14 with 0 pkgs
Spawning worker 15 with 0 pkgs
Workers Finished
Saving Primary metadata
Saving file lists metadata
Saving other metadata
Generating sqlite DBs
Sqlite DBs complete
[root@cdh01 cloudera-repo]# ll
total 1364172
-rwxr-xr-x 1 saicops saicops 14041 Oct 10 14:07 allkeys.asc
-rwxr-xr-x 1 saicops saicops 10479136 Oct 10 14:07 cloudera-manager-agent-6.3.0-1281944.el7.x86_64.rpm
-rwxr-xr-x 1 saicops saicops 1201341068 Oct 10 14:07 cloudera-manager-daemons-6.3.0-1281944.el7.x86_64.rpm
-rwxr-xr-x 1 saicops saicops 11464 Oct 10 14:07 cloudera-manager-server-6.3.0-1281944.el7.x86_64.rpm
-rwxr-xr-x 1 saicops saicops 10996 Oct 10 14:07 cloudera-manager-server-db-2-6.3.0-1281944.el7.x86_64.rpm
-rwxr-xr-x 1 saicops saicops 33887 Oct 10 14:07 manifest.json
-rwxr-xr-x 1 saicops saicops 184988341 Oct 10 14:07 oracle-j2sdk1.8-1.8.0+update181-1.x86_64.rpm
drwxr-xr-x 2 root root 4096 Oct 10 14:08 repodata
-rw-r--r-- 1 saicops saicops 1718 Oct 10 14:07 RPM-GPG-KEY-cloudera
将cloudera-repo整个目录复制到/var/www/html目录下
[root@cdh01 ~]# cp -r cloudera-repo/ /var/www/html/
在/etc/yum.repos.d目录下新建cloudera-manager.repo文件
[root@cdh01 ~]# cd /etc/yum.repos.d/
[root@cdh01 ~]# vim cloudera-manager.repo
在cloudera-manager.repo文件添加如下内容：
[cloudera-manager]
name=Cloudera Manager 6.3.0
baseurl=http://cdh01/cloudera-repo/
gpgkey=http://cdh01/cloudera-repo/RPM-GPG-KEY-cloudera
gpgcheck=1
enabled=1
autorefresh=0
type=rpm-md
保存退出后执行下面的命令
[root@cdh01 yum.repos.d]# yum clean all && yum makecache
安装CDH所需要的依赖包
[root@cdh01 ~]# yum install edhat-lsb portmap mod_ssl openssl-devel python-psycopg2 MySQL-python chkconfig bind-utils libxslt zlib sqlite cyrus-sasl-plain cyrus-sasl-gssapi fuse fuse-libs
如果安装报下面的错
2:postfix-2.10.1-6.el7.x86_64 has missing requires of libmysqlclient.so.18()(64bit)
2:postfix-2.10.1-6.el7.x86_64 has missing requires of libmysqlclient.so.18(libmysqlclient_18)(64bit)
则是缺少依赖，需要安装相关的包，执行下面的操作
[root@cdh01 ~]# wget http://www.percona.com/redir/downloads/Percona-XtraDB-Cluster/5.5.37-25.10/RPM/rhel6/x86_64/Percona-XtraDB-Cluster-shared-55-5.5.37-25.10.756.el6.x86_64.rpm
[root@cdh01 ~]# rpm -ivh Percona-XtraDB-Cluster-shared-55-5.5.37-25.10.756.el6.x86_64.rpm
杀掉supervisor相关的进程并删除supervisor相关的包，因为安装CDH时自带的有，可能会有冲突(所有机器都要进行该操作)
[root@cdh01 ~]# ps -ef |grep supervisor
root 3503 1 0 Aug20 ? 00:05:21 /usr/bin/python /usr/bin/supervisord -c /etc/supervisord.conf
root 6645 6472 0 11:10 pts/0 00:00:00 grep --color=auto supervisor
[root@cdh01 ~]# kill -9 3503
[root@cdh01 ~]# ps -ef |grep supervisor
root 6645 6472 0 11:10 pts/0 00:00:00 grep --color=auto supervisor
[root@cdh01 ~]# rpm -qa |grep supervisor
supervisor-3.1.4-1.el7.noarch
[root@cdh01 ~]# rpm -e supervisor-3.1.4-1.el7.noarch --nodeps
安装CDH
[root@cdh01 ~]# yum install cloudera-manager-daemons cloudera-manager-agent cloudera-manager-server
创建CDH所需要的数据库
create database scm default character set utf8 default collate utf8_general_ci;
grant all on scm.* to 'scm'@'%' identified by 'password';
create database amon default character set utf8 default collate utf8_general_ci;
grant all on amon.* to 'amon'@'%' identified by 'password';
create database rman default character set utf8 default collate utf8_general_ci;
grant all on rman.* to 'rman'@'%' identified by 'password';
create database hue default character set utf8 default collate utf8_general_ci;
grant all on hue.* to 'hue'@'%' identified by 'password';
create database metastore default character set utf8 default collate utf8_general_ci;
grant all on metastore.* to 'hive'@'%' identified by 'password';
create database sentry default character set utf8 default collate utf8_general_ci;
grant all on sentry.* to 'sentry'@'%' identified by 'password';
create database nav default character set utf8 default collate utf8_general_ci;
grant all on nav.* to 'nav'@'%' identified by 'password';
create database oozie default character set utf8 default collate utf8_general_ci;
grant all on oozie.* to 'oozie'@'%' identified by 'password';
flush privileges;
设置CDH数据库
Cloudera Manager Server有一个配置数据库的脚本，执行该脚本就好
如果MySQL数据库与CDH Server在同一台主机上，执行如下命令
[root@cdh01 ~]# /opt/cloudera/cm/schema/scm_prepare_database.sh mysql scm scm
Enter SCM password:
JAVA_HOME=/usr/local/java/
Verifying that we can write to /etc/cloudera-scm-server
Creating SCM configuration file in /etc/cloudera-scm-server
Executing: /usr/local/java//bin/java -cp /usr/share/java/mysql-connector-java.jar:/usr/share/java/oracle-connector-java.jar:/usr/share/java/postgresql-connector-java.jar:/opt/cloudera/cm/schema/../lib/* com.cloudera.enterprise.dbutil.DbCommandExecutor /etc/cloudera-scm-server/db.properties com.cloudera.cmf.db.
Fri Oct 11 14:14:59 CST 2019 WARN: Establishing SSL connection without server's identity verification is not recommended. According to MySQL 5.5.45+, 5.6.26+ and 5.7.6+ requirements SSL connection must be established by default if explicit option isn't set. For compliance with existing applications not using SSL the verifyServerCertificate property is set to 'false'. You need either to explicitly disable SSL by setting useSSL=false, or set useSSL=true and provide truststore for server certificate verification.
[ main] DbCommandExecutor INFO Successfully connected to database.
All done, your SCM database is configured correctly!
如果MySQL数据库与CDH Server不在同一台主机上，执行如下命令
[root@cdh01 ~]# /opt/cloudera/cm/schema/scm_prepare_database.sh mysql -h <mysql-host-ip> --scm-host <cm-server-ip> scm scm
启动CDH Server
[root@cdh01 parcel-repo]# systemctl start cloudera-scm-server
[root@cdh01 parcel-repo]# systemctl start cloudera-scm-agent
[root@cdh01 parcel-repo]# systemctl status cloudera-scm-agent
● cloudera-scm-agent.service - Cloudera Manager Agent Service
Loaded: loaded (/usr/lib/systemd/system/cloudera-scm-agent.service; enabled; vendor preset: disabled)
Active: active (running) since Fri 2019-10-11 14:39:29 CST; 7s ago
Main PID: 23721 (cmagent)
CGroup: /system.slice/cloudera-scm-agent.service
└─23721 /usr/bin/python2 /opt/cloudera/cm-agent/bin/cm agent
Oct 11 14:39:30 cdh01 cm[23721]: [11/Oct/2019 14:39:30 +0000] 23721 MainThread agent INFO Re-using pre-existing directory: /var/run/cloudera-scm-agent/flood
Oct 11 14:39:30 cdh01 cm[23721]: [11/Oct/2019 14:39:30 +0000] 23721 MainThread agent INFO Re-using pre-existing directory: /var/run/cloudera-scm-agent/supervisor/include
Oct 11 14:39:30 cdh01 cm[23721]: [11/Oct/2019 14:39:30 +0000] 23721 MainThread agent INFO Re-using pre-existing directory: /var/run/cloudera-scm-agent/cgroups
Oct 11 14:39:30 cdh01 cm[23721]: [11/Oct/2019 14:39:30 +0000] 23721 MainThread agent INFO Re-using pre-existing directory: /var/run/cloudera-scm-agent/process
Oct 11 14:39:30 cdh01 cm[23721]: [11/Oct/2019 14:39:30 +0000] 23721 MainThread tmpfs INFO Successfully mounted tmpfs at /var/run/cloudera-scm-agent/process
Oct 11 14:39:30 cdh01 cm[23721]: [11/Oct/2019 14:39:30 +0000] 23721 MainThread logging INFO Logging to /var/log/cloudera-scm-agent/cloudera-scm-agent.log
Oct 11 14:39:36 cdh01 cm[23721]: status_server: added process group
Oct 11 14:39:36 cdh01 cm[23721]: flood: added process group
Oct 11 14:39:36 cdh01 cm[23721]: /opt/cloudera/cm-agent/lib/python2.7/site-packages/psutil/_pslinux.py:477: RuntimeWarning: dirty, writeback, mapped, commit_limit memory stats couldn't be...d were set to 0
Oct 11 14:39:36 cdh01 cm[23721]: warnings.warn(msg, RuntimeWarning)
Hint: Some lines were ellipsized, use -l to show in full.
[root@cdh01 parcel-repo]# systemctl status cloudera-scm-server
● cloudera-scm-server.service - Cloudera CM Server Service
Loaded: loaded (/usr/lib/systemd/system/cloudera-scm-server.service; enabled; vendor preset: disabled)
Active: active (running) since Fri 2019-10-11 14:39:23 CST; 20s ago
Main PID: 23566 (java)
CGroup: /system.slice/cloudera-scm-server.service
└─23566 /usr/java/jdk1.8.0_181-cloudera/bin/java -cp .:/usr/share/java/mysql-connector-java.jar:/usr/share/java/oracle-connector-java.jar:/usr/share/java/postgresql-connector-java.jar:lib/* -...
Oct 11 14:39:43 cdh01 cm-server[23566]: Fri Oct 11 14:39:43 CST 2019 WARN: Establishing SSL connection without server's identity verification is not recommended. According to MySQL 5.5.45+...xisting applic
Oct 11 14:39:43 cdh01 cm-server[23566]: Fri Oct 11 14:39:43 CST 2019 WARN: Establishing SSL connection without server's identity verification is not recommended. According to MySQL 5.5.45+...xisting applic
Oct 11 14:39:44 cdh01 cm-server[23566]: Fri Oct 11 14:39:44 CST 2019 WARN: Establishing SSL connection without server's identity verification is not recommended. According to MySQL 5.5.45+...xisting applic
Oct 11 14:39:44 cdh01 cm-server[23566]: Fri Oct 11 14:39:44 CST 2019 WARN: Establishing SSL connection without server's identity verification is not recommended. According to MySQL 5.5.45+...xisting applic
Oct 11 14:39:44 cdh01 cm-server[23566]: Fri Oct 11 14:39:44 CST 2019 WARN: Establishing SSL connection without server's identity verification is not recommended. According to MySQL 5.5.45+...xisting applic
Oct 11 14:39:44 cdh01 cm-server[23566]: Fri Oct 11 14:39:44 CST 2019 WARN: Establishing SSL connection without server's identity verification is not recommended. According to MySQL 5.5.45+...xisting applic
Oct 11 14:39:44 cdh01 cm-server[23566]: Fri Oct 11 14:39:44 CST 2019 WARN: Establishing SSL connection without server's identity verification is not recommended. According to MySQL 5.5.45+...xisting applic
Oct 11 14:39:44 cdh01 cm-server[23566]: Fri Oct 11 14:39:44 CST 2019 WARN: Establishing SSL connection without server's identity verification is not recommended. According to MySQL 5.5.45+...xisting applic
Oct 11 14:39:44 cdh01 cm-server[23566]: Fri Oct 11 14:39:44 CST 2019 WARN: Establishing SSL connection without server's identity verification is not recommended. According to MySQL 5.5.45+...xisting applic
Oct 11 14:39:44 cdh01 cm-server[23566]: Fri Oct 11 14:39:44 CST 2019 WARN: Establishing SSL connection without server's identity verification is not recommended. According to MySQL 5.5.45+...xisting applic
Hint: Some lines were ellipsized, use -l to show in full.
配置集群
本地浏览器上输入http://cdh01:7180，然后进去配置机器即可。
注意：安装集群时，如果报Connection Refused，查看一下httpd服务是否运行正常。
版权声明：本文为gxd520原创文章，遵循CC 4.0 BY-SA版权协议，转载请附上原文出处链接和本声明。