red hat6.5安装教程（0610-6.2.0-如何在Redhat7.4安装CDH6.2-上）
red hat6.5安装教程（0610-6.2.0-如何在Redhat7.4安装CDH6.2-上）PostgreSQL 8.4或更高MariaDB 5.5或更高4.安装CDH集群；请务必注意CDH6的安装前置条件包括如下：MySQL 5.7或更高
1.文档编写目的Cloudera在2019年3月30日，对外宣布正式发布Cloudera Enterprise 6.2，相关介绍可以参考Fayson昨天的文章《0585-Cloudera Enterprise 6.2.0发布》，《0589-Cloudera Manager6.2的新功能》和《0595-CDH6.2的新功能》。本文档Fayson主要描述如何在Redhat7.4安装CDH6.2。CDH6与CDH5的安装步骤一致，主要包括以下四部分：
1.安全前置准备，包括安装操作系统、关闭防火墙、同步服务器时钟等；
2.外部数据库如MySQL安装
3.安装Cloudera Manager；
4.安装CDH集群；
请务必注意CDH6的安装前置条件包括如下：
- 外部数据库支持：
MySQL 5.7或更高
MariaDB 5.5或更高
PostgreSQL 8.4或更高
Oracle 12c或更高
- JDK
Oracle JDK1.8，将不再支持JDK1.7
- 操作系统支持
RHEL 6.8或更高
RHEL 7.2或更高
SLES 12 SP2或更高
Ubuntu 16或更高
- 本次Fayson的测试环境为
1.CM和CDH版本为6.2
2.Redhat7.4
3.JDK1.8.0_181
hostname配置
/etc/hostname文件如下：
ip-172-31-6-83.ap-southeast-1.compute.internal
或者你可以通过命令修改立即生效
[root@ip-172-31-6-83 ~]$ hostnamectl set-hostname ip-172-31-6-83.ap-southeast-1.compute.internal
- hosts配置
/etc/hosts文件如下:
127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4 ::1 localhost localhost.localdomain localhost6 localhost6.localdomain6 172.31.6.83 ip-172-31-6-83.ap-southeast-1.compute.internal 172.31.12.142 ip-172-31-12-142.ap-southeast-1.compute.internal 172.31.9.113 ip-172-31-9-113.ap-southeast-1.compute.internal 172.31.4.105 ip-172-31-4-105.ap-southeast-1.compute.internal
以上两步操作，在集群中其它节点做相应配置。确认需要安装的4台主机的hosts文件：
在所有节点执行setenforce 0 命令，此处使用批处理shell执行：
[root@ip-172-31-6-83 shell]# sh ssh_do_all.sh node.list "setenforce 0"
集群所有节点修改/etc/selinux/config文件如下：
# This file controls the state of SELinux on the system. # SELINUX= can take one of these three values: # enforcing - SELinux security policy is enforced. # permissive - SELinux prints warnings instead of enforcing. # disabled - No SELinux policy is loaded. SELINUX=disabled # SELINUXTYPE= can take one of three two values: # targeted - Targeted processes are protected # minimum - Modification of targeted policy. Only selected processes are protected. # mls - Multi Level Security protection. SELINUXTYPE=targeted
集群所有节点执行 systemctl stop命令，此处通过shell批量执行命令如下：
[root@ip-172-31-6-83 shell]# sh ssh_do_all.sh node.list "systemctl stop firewalld" [root@ip-172-31-6-83 shell]# sh ssh_do_all.sh node.list "systemctl disable firewalld" [root@ip-172-31-6-83 shell]# sh ssh_do_all.sh node.list "systemctl status firewalld"
在Redhat7.x的操作系统上，已经默认的安装了chrony，我们这里先卸载chrony，然后安装ntp。使用ntp来配置各台机器的时钟同步，将cm（172.31.6.83）服务作为本地ntp服务器，其它3台服务器与其保持同步。
1.所有机器卸载chrony
[root@ip-172-31-6-83 shell]# sh ssh_do_all.sh node.list "yum -y remove chrony"
2.所有机器安装ntp
[root@ip-172-31-6-83 shell]# sh ssh_do_all.sh node.list "yum -y install ntp"
3.cm机器配置时钟与自己同步
[root@ ip-172-31-6-83 shell]# vim /etc/ntp.conf #server 0.rhel.pool.ntp.org iburst #server 1.rhel.pool.ntp.org iburst #server 2.rhel.pool.ntp.org iburst #server 3.rhel.pool.ntp.org iburst server 127.127.1.0 # local clock fudge 127.127.1.0 stratum 10
4.集群其它节点，配置找cm机器去同步
#server 0.rhel.pool.ntp.org iburst #server 1.rhel.pool.ntp.org iburst #server 2.rhel.pool.ntp.org iburst #server 3.rhel.pool.ntp.org iburst server 172.31.6.83
5.重启所有机器的ntp服务
[root@ ip-172-31-6-83 shell]# sh ssh_do_all.sh node.list "systemctl restart ntpd" [root@ ip-172-31-6-83 shell]# sh ssh_do_all.sh node.list "systemctl status ntpd"
6.验证始终同步，在所有节点执行ntpq -p命令，如下使用脚本批量执行
[root@ ip-172-31-6-83 shell]# sh ssh_do_all.sh node.list "ntpq -p"
左边出现*号表示同步成功。
2.5.设置swap所有节点执行
[root@ ip-172-31-6-83 shell]# sh ssh_do_all.sh node.list "echo vm.swappiness = 1 >> /etc/sysctl.conf" [root@ ip-172-31-6-83 shell]#
所有节点执行：
[root@ ip-172-31-6-83 shell]# sh ssh_do_all.sh node.list "echo never > /sys/kernel/mm/transparent_hugepage/defrag " [root@ ip-172-31-6-83 shell]# sh ssh_do_all.sh node.list "echo never > /sys/kernel/mm/transparent_hugepage/enabled"
设置开机自关闭
将如下脚本添加到/etc/rc.d/rc.local文件中
if test -f /sys/kernel/mm/transparent_hugepage/enabled; then echo never > /sys/kernel/mm/transparent_hugepage/enabled fi if test -f /sys/kernel/mm/transparent_hugepage/defrag; then echo never > /sys/kernel/mm/transparent_hugepage/defrag fi
同步到所有节点
Fayson用的是AWS的环境，这步是可以省略的，放在这里供物理机部署的兄弟们参考。
- 挂载操作系统iso文件
[ec2-user@ip-172-31-2-159 ~]$ sudo mkdir /media/DVD1 [ec2-user@ip-172-31-2-159 ~]$ sudo mount -o loop CentOS-7-x86_64-DVD-1611.iso /media/DVD1/
- 配置操作系统repo
[ec2-user@ip-172-31-2-159 ~]$ sudo vim /etc/yum.repos.d/local_os.repo [local_iso] name=CentOS-$releasever - Media baseurl=file:///media/DVD1 gpgcheck=0 enabled=1 [ec2-user@ip-172-31-2-159 ~]$ sudo yum repolist 2.8.安装http服务
- 安装httpd服务
[ec2-user@ip-172-31-2-159 ~]$ sudo yum -y install httpd
- 启动httpd服务
[ec2-user@ip-172-31-2-159 ~]$ sudo systemctl start httpd
- 安装完httpd后，重新制作操作系统repo，换成http的方式方便其它服务器也可以访问
[ec2-user@ip-172-31-2-159 ~]$ sudo mkdir /var/www/html/iso [ec2-user@ip-172-31-2-159 ~]$ sudo scp -r /media/DVD1/* /var/www/html/iso/ [ec2-user@ip-172-31-2-159 ~]$ sudo vim /etc/yum.repos.d/os.repo [osrepo] name=os_repo baseurl=http://172.31.2.159/iso/ enabled=true gpgcheck=false [ec2-user@ip-172-31-2-159 ~]$ sudo yum repolist
1.修改/etc/httpd/conf/httpd.conf配置文件，在<IfModule mime_module>中修改以下内容
AddType application/x-gzip .gz .tgz .parcel
2.保存httpd.conf的修改，并重启httpd服务
[root@ip-172-31-6-83 java]# systemctl restart httpd
1.安装MariaDB
[root@ ip-172-31-6-83 ~]# yum -y install mariadb [root@ ip-172-31-6-83 ~]# yum -y install mariadb-server
2.启动并配置MariaDB
[root@ ip-172-31-6-83 ~]# systemctl start mariadb [root@ ip-172-31-6-83 ~]# /usr/bin/mysql_secure_installation NOTE: RUNNING ALL PARTS OF THIS SCRIPT IS RECOMMENDED FOR ALL MariaDB SERVERS IN PRODUCTION USE! PLEASE READ EACH STEP CAREFULLY! In order to log into MariaDB to secure it we'll need the current password for the root user. If you've just installed MariaDB and you haven't set the root password yet the password will be blank so you should just press enter here. Enter current password for root (enter for none): OK successfully used password moving on... Setting the root password ensures that nobody can log into the MariaDB root user without the proper authorisation. Set root password? [Y/n] Y New password: Re-enter new password: Password updated successfully! Reloading privilege tables.. ... Success! By default a MariaDB installation has an anonymous user allowing anyone to log into MariaDB without having to have a user account created for them. This is intended only for testing and to make the installation go a bit smoother. You should remove them before moving into a production environment. Remove anonymous users? [Y/n] Y ... Success! Normally root should only be allowed to connect from 'localhost'. This ensures that someone cannot guess at the root password from the network. Disallow root login remotely? [Y/n] n ... skipping. By default MariaDB comes with a database named 'test' that anyone can access. This is also intended only for testing and should be removed before moving into a production environment. Remove test database and access to it? [Y/n] Y - Dropping test database... ... Success! - Removing privileges on test database... ... Success! Reloading the privilege tables will ensure that all changes made so far will take effect immediately. Reload privilege tables now? [Y/n] Y ... Success! Cleaning up... All done! If you've completed all of the above steps your MariaDB installation should now be secure. Thanks for using MariaDB!
3.建立CM，Hive等需要的表
[root@ip-172-31-6-83 ~]# mysql -u root -p Enter password: Welcome to the MariaDB monitor. Commands end with ; or \g. Your MariaDB connection id is 9 Server version: 5.5.56-MariaDB MariaDB Server Copyright (c) 2000 2017 Oracle MariaDB Corporation Ab and others. Type 'help;' or '\h' for help. Type '\c' to clear the current input statement. MariaDB [(none)]> create database metastore default character set utf8; CREATE USER 'hive'@'%' IDENTIFIED BY 'password'; GRANT ALL PRIVILEGES ON metastore. * TO 'hive'@'%'; FLUSH PRIVILEGES; create database cm default character set utf8; CREATE USER 'cm'@'%' IDENTIFIED BY 'password'; GRANT ALL PRIVILEGES ON cm. * TO 'cm'@'%'; FLUSH PRIVILEGES; create database am default character set utf8; CREATE USER 'am'@'%' IDENTIFIED BY 'password'; GRANT ALL PRIVILEGES ON am. * TO 'am'@'%'; FLUSH PRIVILEGES; create database rm default character set utf8; CREATE USER 'rm'@'%' IDENTIFIED BY 'password'; GRANT ALL PRIVILEGES ON rm. * TO 'rm'@'%'; FLUSH PRIVILEGES; create database hue default character set utf8; CREATE USER 'hue'@'%' IDENTIFIED BY 'password'; GRANT ALL PRIVILEGES ON hue. * TO 'hue'@'%'; FLUSH PRIVILEGES; create database oozie default character set utf8; CREATE USER 'oozie'@'%' IDENTIFIED BY 'password'; GRANT ALL PRIVILEGES ON oozie. * TO 'oozie'@'%'; FLUSH PRIVILEGES; create database sentry default character set utf8; CREATE USER 'sentry'@'%' IDENTIFIED BY 'password'; GRANT ALL PRIVILEGES ON sentry. * TO 'sentry'@'%'; FLUSH PRIVILEGES; create database nav_ms default character set utf8; CREATE USER 'nav_ms'@'%' IDENTIFIED BY 'password'; GRANT ALL PRIVILEGES ON nav_ms. * TO 'nav_ms'@'%'; FLUSH PRIVILEGES; create database nav_as default character set utf8; CREATE USER 'nav_as'@'%' IDENTIFIED BY 'password'; GRANT ALL PRIVILEGES ON nav_as. * TO 'nav_as'@'%'; FLUSH PRIVILEGES;
- 安装JDBC驱动
[root@ip-172-31-6-83 shell]# mkdir -p /usr/share/java/ [root@ip-172-31-6-83 ~]# mv mysql-connector-java-5.1.34.jar /usr/share/java/ [root@ip-172-31-6-83 ~]# cd /usr/share/java [root@ip-172-31-6-83 java]# chmod 777 mysql-connector-java-5.1.34.jar [root@ip-172-31-6-83 java]# ln -s mysql-connector-java-5.1.34.jar mysql-connector-java.jar [root@ip-172-31-6-83 java]# ll total 940 -rwxrwxrwx. 1 root root 960372 May 16 15:53 mysql-connector-java-5.1.34.jar lrwxrwxrwx. 1 root root 31 May 16 15:53 mysql-connector-java.jar -> mysql-connector-java-5.1.34.jar
1.下载CM6.2的安装包，地址为：
https://archive.cloudera.com/cm6/6.2.0/redhat7/yum/rpmS/x86_64/cloudera-manager-agent-6.2.0-968826.el7.x86_64.rpm https://archive.cloudera.com/cm6/6.2.0/redhat7/yum/RPMS/x86_64/cloudera-manager-daemons-6.2.0-968826.el7.x86_64.rpm https://archive.cloudera.com/cm6/6.2.0/redhat7/yum/RPMS/x86_64/cloudera-manager-server-6.2.0-968826.el7.x86_64.rpm https://archive.cloudera.com/cm6/6.2.0/redhat7/yum/RPMS/x86_64/cloudera-manager-server-db-2-6.2.0-968826.el7.x86_64.rpm https://archive.cloudera.com/cm6/6.2.0/redhat7/yum/RPMS/x86_64/enterprise-debuginfo-6.2.0-968826.el7.x86_64.rpm https://archive.cloudera.com/cm6/6.2.0/redhat7/yum/RPMS/x86_64/oracle-j2sdk1.8-1.8.0 update181-1.x86_64.rpm https://archive.cloudera.com/cm6/6.2.0/allkeys.asc
2.下载CDH6.2的安装包，地址为：
https://archive.cloudera.com/cdh6/6.2.0/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373-el7.parcel https://archive.cloudera.com/cdh6/6.2.0/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373-el7.parcel.sha1 https://archive.cloudera.com/cdh6/6.2.0/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373-el7.parcel.sha256 https://archive.cloudera.com/cdh6/6.2.0/parcels/manifest.json
3.将Cloudera Manager安装需要的6个rpm包以及一个asc文件下载到本地，放在同一目录，执行createrepo命令生成rpm元数据。
[root@ip-172-31-6-83 cm6.2]# ll total 1364096 -rw-r--r-- 1 root root 14041 Mar 28 17:13 allkeys.asc -rw-r--r-- 1 root root 10215488 Mar 28 17:13 cloudera-manager-agent-6.2.0-968826.el7.x86_64.rpm -rw-r--r-- 1 root root 1187380436 Mar 28 17:13 cloudera-manager-daemons-6.2.0-968826.el7.x86_64.rpm -rw-r--r-- 1 root root 9984 Mar 28 17:13 cloudera-manager-server-6.2.0-968826.el7.x86_64.rpm -rw-r--r-- 1 root root 10992 Mar 28 17:13 cloudera-manager-server-db-2-6.2.0-968826.el7.x86_64.rpm -rw-r--r-- 1 root root 14200108 Mar 28 17:13 enterprise-debuginfo-6.2.0-968826.el7.x86_64.rpm -rw-r--r-- 1 root root 184988341 Mar 28 17:13 oracle-j2sdk1.8-1.8.0 update181-1.x86_64.rpm [root@ip-172-31-6-83 cm6.2]# createrepo . Spawning worker 0 with 2 pkgs Spawning worker 1 with 2 pkgs Spawning worker 2 with 1 pkgs Spawning worker 3 with 1 pkgs Workers Finished Saving Primary metadata Saving file lists metadata Saving other metadata Generating sqlite DBs Sqlite DBs complete
4.配置Web服务器
将上述cdh6.2/cm6.2目录移动到/var/www/html目录下 使得用户可以通过HTTP访问这些rpm包。
[root@ip-172-31-6-83 ~]# mv cm6.2/ cdh6.2/ /var/www/html/ [root@ip-172-31-6-83 ~]# cd /var/www/html/ [root@ip-172-31-6-83 html]# ll *6.2
验证浏览器能否正常访问
5.制作Cloudera Manager的repo源
[root@ip-172-31-6-83 ~]# vim /etc/yum.repos.d/cm.repo [cmrepo] name = cm_repo baseurl = baseurl=http://172.31.6.83/cm6.2 enable = true gpgcheck = false [root@ip-172-31-6-83 yum.repos.d]# yum repolist Loaded plugins: amazon-id rhui-lb search-disabled-repos repo id repo name status cmrepo cm_repo 6 rhui-REGION-client-config-server-7/x86_64 Red Hat Update Infrastructure 2.0 Client Configuration Server 7 2 rhui-REGION-rhel-server-releases/7Server/x86_64 Red Hat Enterprise Linux Server 7 (RPMs) 23 947 rhui-REGION-rhel-server-rh-common/7Server/x86_64 Red Hat Enterprise Linux Server 7 RH Common (RPMs) 235 repolist: 24 190
6.验证安装JDK
[root@ip-172-31-6-83 cm6.2]# yum -y install oracle-j2sdk1.8-1.8.0 update181-1.x86_64
1.通过yum安装Cloudera Manager Server
yum -y install cloudera-manager-server
2.初始化数据库
[root@ip-172-31-6-83 cm6.2]# /opt/cloudera/cm/schema/scm_prepare_database.sh mysql cm cm password JAVA_HOME=/usr/java/jdk1.8.0_181-cloudera Verifying that we can write to /etc/cloudera-scm-server Creating SCM configuration file in /etc/cloudera-scm-server Executing: /usr/java/jdk1.8.0_181-cloudera/bin/java -cp /usr/share/java/mysql-connector-java.jar:/usr/share/java/oracle-connector-java.jar:/usr/share/java/postgresql-connector-java.jar:/opt/cloudera/cm/schema/../lib/* com.cloudera.enterprise.dbutil.DbCommandExecutor /etc/cloudera-scm-server/db.properties com.cloudera.cmf.db. [ main] DbCommandExecutor INFO Successfully connected to database. All done your SCM database is configured correctly! [root@ip-172-31-6-83 cm6.2]#
3.启动Cloudera Manager Server
[root@ ip-172-31-6-83 ~]# systemctl start cloudera-scm-server
4.检查端口是否监听
[root@ip-172-31-6-83 cm6.2]# netstat -lnpt | grep 7180 tcp 0 0 0.0.0.0:7180 0.0.0.0:* LISTEN 1308/java
5.通过http://cm_ip:7180/cmf/login访问CM
