在 阿里云ECS 上安装 CDH6.2 standalong

在 阿里云ECS 上安装 CDH6.2

0. pre Install

0.1 ECS 配置
基本配置至少 4核 16GB，比较烧 内存

8核 + 64GB 100GB SSD 5M 宽带

操作系统看 CDH 官网支持，推荐 RHEL6, RHEL7, CentOs7，总之不推荐 Ali Linux 或者 Amazon Linux

centos7.7

安全组

保证本地可以连接 7180, 22 端口

密钥，win系统需要再转换为 pub

xxx.pem密钥
xxx.pem.pub 密钥

CDH 6.2

下载路径
公众号【大数据1024】输入 CDH
文件在 2.1 里面列出来了

配置的软件

SecureCRT
FileZilla

1. 系统环境准备

1.1 设置 ssh 生存时间
避免远程连接时一会不操作就断开连接

vi /etc/ssh/sshd_configClientAliveInterval 30ClientAliveCountMax 1800

1.2 设置 时区

timedatectl set-timezone Asia/Shanghai

1.3 设置 主机名

vi /etc/hostnameStock-DMP source /etc/profile

1.4 安装 基础包

yum -y install wget yum -y install lrzsz # SecureCRT 上传下载小文件利器 yum -y install zip unzip yum -y install vim

1.5 关闭 防火墙
阿里云 防火墙 默认是关闭的

systemctl status firewalld systemctl stop firewalld systemctl disable firewalld

1.6 关闭 SELINUX
阿里云 SELINUX 默认是关闭的

vim /etc/selinux/configSELINUX=disabled

1.7 修改 Linux swapping 和虚拟内存
我的机器默认 vm.swappiness = 0
swappiness = 0 表示最大限度使用物理内存，然后才是 swap 空间。有兴趣自行搜索调整到合适参数。

vi /etc/sysctl.confvm.swappiness = 0vm.max_map_count=655360 sysctl -p

1.8 禁用 THP (Transparent HugePages)

对于数据库应用，不推荐使用 THP。主要的原因是这类数据库大部分访问内存的方式是分散的，并不是访问连续的页面，而这样的访问模式,就会造成内存的碎片化。

# 查看 THP 状态, Always 表示启用 [root@Stock-DMP soft]# cat /sys/kernel/mm/transparent_hugepage/defrag [always] madvise never # 禁用 THP 并添加到配置文件 [root@Stock-DMP soft]# echo never > /sys/kernel/mm/transparent_hugepage/defrag [root@Stock-DMP soft]# echo never > /sys/kernel/mm/transparent_hugepage/enabled [root@Stock-DMP soft]# echo "echo never > /sys/kernel/mm/transparent_hugepage/defrag" >> /etc/rc.local [root@Stock-DMP soft]# echo "echo never > /sys/kernel/mm/transparent_hugepage/enabled" >> /etc/rc.local [root@Stock-DMP soft]# chmod +x /etc/rc.d/rc.local # 查看 THP 状态, Never 表示禁用 [root@Stock-DMP soft]# cat /etc/rc.d/rc.local always madvise [never]

1.9 重启服务器

reboot -h

2. 安装环境

2.1 通过 FileZilla 上传需要的文件
文件大致如下

mkdir -p /soft [root@Stock-DMP CDH6.2]# ll total 2754532 -rw-r--r-- 1 root root 2093332003 Jan 7 16:48 CDH-6.2.1-1.cdh6.2.1.p0.1425774-el7.parcel drwxr-xr-x 2 root root 4096 Jan 7 16:47 cloudera-repos -rw-r--r-- 1 root root 95840 Jan 7 16:33 createrepo-0.9.9-28.el7.noarch.rpm -rw-r--r-- 1 root root 541163520 Jan 7 16:37 mysql-5.7.31-1.el7.x86_64.rpm-bundle.tar -rw-r--r-- 1 root root 1007502 Jan 7 16:37 mysql-connector-java-5.1.47.jar -rw-r--r-- 1 root root 184988341 Jan 7 16:38 oracle-j2sdk1.8-1.8.0+update181-1.x86_64.rpm -rw-r--r-- 1 root root 32084 Jan 7 16:38 python-deltarpm-3.6-3.el7.x86_64.rpm [root@Stock-DMP CDH6.2]# ll cloudera-repos/ total 1181628 -rw-r--r-- 1 root root 14041 Jan 7 16:38 allkeys.asc -rw-r--r-- 1 root root 10215608 Jan 7 16:38 cloudera-manager-agent-6.2.1-1426065.el7.x86_64.rpm -rw-r--r-- 1 root root 1199718256 Jan 7 16:47 cloudera-manager-daemons-6.2.1-1426065.el7.x86_64.rpm -rw-r--r-- 1 root root 9988 Jan 7 16:47 cloudera-manager-server-6.2.1-1426065.el7.x86_64.rpm -rw-r--r-- 1 root root 10996 Jan 7 16:47 cloudera-manager-server-db-2-6.2.1-1426065.el7.x86_64.rpm -rw-r--r-- 1 root root 1718 Jan 7 16:47 RPM-GPG-KEY-cloudera [root@Stock-DMP CDH6.2]# pwd /soft/CDH6.2

2.2 安装 JDK
CDH 6.2 支持 jdk1.8
其他版本注意版本对应关系
关于rpm, 可以参考 Linux RPM 命令参数解释

[root@Stock-DMP CDH6.2]# rpm -ivh oracle-j2sdk1.8-1.8.0+update181-1.x86_64.rpm warning: oracle-j2sdk1.8-1.8.0+update181-1.x86_64.rpm: Header V3 RSA/SHA256 Signature, key ID b0b19c9f: NOKEY Preparing... ################################# [100%] Updating / installing...1:oracle-j2sdk1.8-1.8.0+update181-1################################# [100%] [root@Stock-DMP CDH6.2]# vi /etc/profile export JAVA_HOME=/usr/java/jdk1.8.0_181-cloudera export PATH=.:$PATH:$JAVA_HOME/bin [root@Stock-DMP CDH6.2]# source /etc/profile [root@Stock-DMP CDH6.2]# java -version java version "1.8.0_181" Java(TM) SE Runtime Environment (build 1.8.0_181-b13) Java HotSpot(TM) 64-Bit Server VM (build 25.181-b13, mixed mode) [root@Stock-DMP CDH6.2]# which java /usr/java/jdk1.8.0_181-cloudera/bin/java

2.3 安装 Mysql
卸载 centos7 原生的 mariadb

[root@Stock-DMP CDH6.2]# rpm -qa|grep mariadb mariadb-libs-5.5.64-1.el7.x86_64 [root@Stock-DMP CDH6.2]# rpm -e --nodeps mariadb-libs-5.5.64-1.el7.x86_64

执行以下命令

[root@Stock-DMP CDH6.2]# tar -xvf mysql-5.7.31-1.el7.x86_64.rpm-bundle.tar [root@Stock-DMP CDH6.2]# rpm -ivh mysql-community-common-5.7.31-1.el7.x86_64.rpm --force --nodeps [root@Stock-DMP CDH6.2]# rpm -ivh mysql-community-libs-5.7.31-1.el7.x86_64.rpm --force --nodeps [root@Stock-DMP CDH6.2]# rpm -ivh mysql-community-client-5.7.31-1.el7.x86_64.rpm --force --nodeps [root@Stock-DMP CDH6.2]# rpm -ivh mysql-community-server-5.7.31-1.el7.x86_64.rpm --force --nodeps [root@Stock-DMP CDH6.2]# rpm -ivh mysql-community-libs-compat-5.7.31-1.el7.x86_64.rpm --force --nodeps # 验证 [root@Stock-DMP CDH6.2]# rpm -qa|grep mysql mysql-community-libs-5.7.31-1.el7.x86_64 mysql-community-common-5.7.31-1.el7.x86_64 mysql-community-client-5.7.31-1.el7.x86_64 mysql-community-libs-compat-5.7.31-1.el7.x86_64 mysql-community-server-5.7.31-1.el7.x86_64

如果遇到以下 warning 是由于 yum 安装了旧版本的 GPG keys 所造成，从 rpm 版本 4.1 后，在安装或升级软件包时会自动检查软件包的签名 参考资料

warning: mysql-community-common-5.7.31-1.el7.x86_64.rpm: Header V3 DSA/SHA1 Signature, key ID 5072e1f5: NOKEY

初始化 Mysql 并获取初始密码 farBa5Iys&PL

[root@Stock-DMP CDH6.2]# yum install -y libaio [root@Stock-DMP CDH6.2]# mysqld --initialize --user=mysql [root@Stock-DMP CDH6.2]# cat /var/log/mysqld.log | tail -1 2022-01-07T09:47:06.334066Z 1 [Note] A temporary password is generated for root@localhost: farBa5Iys&PL

登录 mysql

# 启动 mysql 服务 [root@Stock-DMP CDH6.2]# systemctl start mysqld.service # 设置 mysql 服务开机启动 [root@Stock-DMP CDH6.2]# systemctl enable mysqld.service # 重启 mysql 服务 [root@Stock-DMP CDH6.2]# systemctl restart mysqld.service # 登录 mysql 并将 root 用户密码设置为 admin [root@Stock-DMP CDH6.2]# mysql -uroot -p Enter password: Welcome to the MySQL monitor. Commands end with ; or \g. Your MySQL connection id is 2 Server version: 5.7.31Copyright (c) 2000, 2020, Oracle and/or its affiliates. All rights reserved.Oracle is a registered trademark of Oracle Corporation and/or its affiliates. Other names may be trademarks of their respective owners.Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.mysql> alter user user() identified by "admin"; Query OK, 0 rows affected (0.00 sec)mysql>

2.3.1 创建 CM 需要的数据库
在 3.6 配置数据库 需要使用

CREATE DATABASE scm DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci; CREATE DATABASE amon DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci; CREATE DATABASE rman DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci; CREATE DATABASE hue DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci; CREATE DATABASE metastore DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci; CREATE DATABASE sentry DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci; CREATE DATABASE nav DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci; CREATE DATABASE navms DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci; CREATE DATABASE oozie DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci; CREATE DATABASE hive DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;CREATE USER 'hive'@'%' IDENTIFIED BY 'hive@2021'; GRANT ALL ON *.* TO 'hive'@'%'; CREATE DATABASE hive; flush privileges; CREATE USER 'scm'@'%' IDENTIFIED BY 'scm@2021'; GRANT ALL ON *.* TO 'scm'@'%'; CREATE DATABASE scm; flush privileges; CREATE USER 'oozie'@'%' IDENTIFIED BY 'oozie@2021'; GRANT ALL ON *.* TO 'oozie'@'%'; CREATE DATABASE oozie; flush privileges; CREATE USER 'hue'@'%' IDENTIFIED BY 'hue@2021'; GRANT ALL ON *.* TO 'hue'@'%'; CREATE DATABASE hue; flush privileges;

更改 /etc/my.cnf 配置

vim /etc/my.cnf [mysqld] basedir = /usr/local/mysql datadir = /var/lib/mysql port = 3306 socket=/var/lib/mysql/mysql.sock character-set-server=utf8transaction-isolation = READ-COMMITTED # Disabling symbolic-links is recommended to prevent assorted security risks; # to do so, uncomment this line: symbolic-links = 0server_id=1 max-binlog-size = 500M log_bin=/var/lib/mysql/mysql_binary_log #binlog_format = mixed binlog_format = Row expire-logs-days = 14max_connections = 550 read_buffer_size = 2M read_rnd_buffer_size = 16M sort_buffer_size = 8M join_buffer_size = 8M# InnoDB settings innodb_file_per_table = 1 innodb_flush_log_at_trx_commit = 2 innodb_log_buffer_size = 64M innodb_buffer_pool_size = 4G innodb_thread_concurrency = 8 innodb_flush_method = O_DIRECT innodb_log_file_size = 512M[client] default-character-set=utf8 socket=/var/lib/mysql/mysql.sock[mysql] default-character-set=utf8 socket=/var/lib/mysql/mysql.sock[mysqld_safe] log-error=/var/log/mysqld.log pid-file=/var/run/mysqld/mysqld.pidsql_mode=STRICT_ALL_TABLES skip-ssl

MySQL 连接器配置
移动到新建目录下 并改为 mysql-connector-java.jar

[root@Stock-DMP CDH6.2]# mkdir -p /usr/share/java [root@Stock-DMP CDH6.2]# mv mysql-connector-java-5.1.47.jar /usr/share/java/mysql-connector-java.jar [root@Stock-DMP CDH6.2]# ll /usr/share/java total 984 -rw-r--r-- 1 root root 1007502 Jan 7 16:37 mysql-connector-java.jar

2.4 安装 第三方依赖

yum -y install chkconfig python bind-utils psmisc libxslt zlib sqlite cyrus-sasl-plain cyrus-sasl-gssapi fuse fuse-libs redhat-lsb httpd mod_ssl portmap /lib/lsb/init-functions openssl-devel python-psycopg2 MySQL-python yum -y install nfs-utils rpcbind

2.5 Cloudera Manager 相关文件
httpd 服务

[root@Stock-DMP CDH6.2]# yum install -y httpd [root@Stock-DMP CDH6.2]# systemctl start httpd.service [root@Stock-DMP CDH6.2]# systemctl enable httpd.service

移动文件到指定路径

[root@Stock-DMP soft]# mkdir -p /var/www/html [root@Stock-DMP soft]# mv /soft/cloudera-repos /var/www/html [root@Stock-DMP soft]# ll /var/www/html/cloudera-repos/ total 1181628 -rw-r--r-- 1 root root 14041 Jan 7 16:38 allkeys.asc -rw-r--r-- 1 root root 10215608 Jan 7 16:38 cloudera-manager-agent-6.2.1-1426065.el7.x86_64.rpm -rw-r--r-- 1 root root 1199718256 Jan 7 16:47 cloudera-manager-daemons-6.2.1-1426065.el7.x86_64.rpm -rw-r--r-- 1 root root 9988 Jan 7 16:47 cloudera-manager-server-6.2.1-1426065.el7.x86_64.rpm -rw-r--r-- 1 root root 10996 Jan 7 16:47 cloudera-manager-server-db-2-6.2.1-1426065.el7.x86_64.rpm -rw-r--r-- 1 root root 1718 Jan 7 16:47 RPM-GPG-KEY-cloudera

浏览器可以 访问（IP 地址为阿里云公有 IP）

2.6 CDH Parcel 相关文件

mkdir -p /opt/cloudera/parcel-repo mv /soft/CDH6.2/CDH-6.2.1-1.cdh6.2.1.p0.1425774-el7.parcel /opt/cloudera/parcel-repo/ sha1sum CDH-6.2.1-1.cdh6.2.1.p0.1425774-el7.parcel | awk '{print $1}' > CDH-6.2.1-1.cdh6.2.1.p0.1425774-el7.parcel.sha useradd --system --home=/opt/cloudera-manager/cm-6.2.1/run/cloudera-scm-server --no-create-home --shell=/bin/false --comment "Cloudera SCM User" cloudera-scm chown -R cloudera-scm:cloudera-scm /opt/cloudera/parcel-repo/

2.7 Cloudera Manager 安装
按顺序安装

[root@Stock-DMP cloudera-repos]# yum -y localinstall http://localhost/cloudera-repos/cloudera-manager-daemons-6.2.1-1426065.el7.x86_64.rpm [root@Stock-DMP cloudera-repos]# yum -y localinstall http://localhost/cloudera-repos/cloudera-manager-server-6.2.1-1426065.el7.x86_64.rpm [root@Stock-DMP cloudera-repos]# yum -y localinstall http://localhost/cloudera-repos/cloudera-manager-agent-6.2.1-1426065.el7.x86_64.rpm

初始化 Cloudera Manager Database

[root@Stock-DMP cloudera-repos]# /opt/cloudera/cm/schema/scm_prepare_database.sh mysql scm root admin JAVA_HOME=/usr/java/jdk1.8.0_181-cloudera Verifying that we can write to /etc/cloudera-scm-server Creating SCM configuration file in /etc/cloudera-scm-server Executing: /usr/java/jdk1.8.0_181-cloudera/bin/java -cp /usr/share/java/mysql-connector-java.jar:/usr/share/java/oracle-connector-java.jar:/usr/share/java/postgresql-connector-java.jar:/opt/cloudera/cm/schema/../lib/* com.cloudera.enterprise.dbutil.DbCommandExecutor /etc/cloudera-scm-server/db.properties com.cloudera.cmf.db. log4j:ERROR Could not find value for key log4j.appender.A log4j:ERROR Could not instantiate appender named "A". Fri Jan 07 18:58:59 CST 2022 WARN: Establishing SSL connection without server's identity verification is not recommended. According to MySQL 5.5.45+, 5.6.26+ and 5.7.6+ requirements SSL connection must be established by default if explicit option isn't set. For compliance with existing applications not using SSL the verifyServerCertificate property is set to 'false'. You need either to explicitly disable SSL by setting useSSL=false, or set useSSL=true and provide truststore for server certificate verification. [2022-01-07 18:58:59,939] INFO 0[main] - com.cloudera.enterprise.dbutil.DbCommandExecutor.testDbConnection(DbCommandExecutor.java) - Successfully connected to database. All done, your SCM database is configured correctly!

启动 Cloudera Manager

[root@Stock-DMP cloudera-repos]# systemctl start cloudera-scm-server [root@Stock-DMP cloudera-repos]# systemctl enable cloudera-scm-server [root@Stock-DMP cloudera-repos]# systemctl start cloudera-scm-agent [root@Stock-DMP cloudera-repos]# systemctl enable cloudera-scm-agent

通过浏览器访问，初始用户名 admin，密码 admin

3. 配置Cloudera Manager

3.1 欢迎页
3.2 版本选择
3.3 集群名称
3.4 集群名称
选择当前管理的主机，不然可能无法成功安装
3.5 依照需求配置服务
后面可以加所以一开始不必要配置很多
3.6 配置数据库
在 2.3.1 创建 CM 需要的数据库 操作过
3.7 启动集群
这里可能会遇到一些问题，可能就是之前配置出了一些问题
3.8 完成
集群仍然有警告信息，由于 HDFS 副本不足和机器数量不足决定的

总结

以上是生活随笔为你收集整理的在 阿里云ECS 上安装 CDH6.2 standalong的全部内容，希望文章能够帮你解决所遇到的问题。

如果觉得生活随笔网站内容还不错，欢迎将生活随笔推荐给好友。