歐拉操作系統下安裝hadoop集群

背景：歐拉操作系統下安裝CDH集群的時候，需要安裝python2.7.5，但是本身歐拉系統對python2的支持可能沒有那么好，所以考慮搭建原生的hadoop集群。

基礎環境如下

組件名稱	組件版本
歐拉	VERSION=“22.03 (LTS-SP4)”
jdk	openjdk version “1.8.0_44”
mysql	8.0.42
hadoop
hive

一、jdk的安裝

openjdk的下載地址

https://jdk.java.net/java-se-ri/8-MR6

二、mysql的安裝

mysql安裝包的下載地址
https://dev.mysql.com/downloads/mysql/

三、hadoop的安裝

hadoop安裝的前置條件是系統中已經有Java的環境
還需要將免密登錄配置好

創建Hadoop用戶

#創建Hadoop的用戶
sudo useradd hadoop#設置Hadoop用戶的密碼
sudo passwd hadoop   #設置Hadoop的用戶有sudo權限            
sudo usermod -aG sudo hadoop

解壓Hadoop的安裝包

#解壓這個壓縮包到/opt的目錄下，這里最好別放在/root下
sudo tar -xzf hadoop-3.4.1.tar.gz -C /opt#給這個解壓完的目錄，修改一個目錄名
sudo mv /opt/hadoop-3.4.1 /opt/hadoop#設置這個路徑的用戶、用戶組
sudo chown -R hadoop:hadoop /opt/hadoop

創建環境變量在配置文件中

#將配置寫入到配置文件中
echo 'export HADOOP_HOME=/opt/hadoop' | sudo tee -a /etc/profile
echo 'export PATH=$PATH:$HADOOP_HOME/bin:$HADOOP_HOME/sbin' | sudo tee -a /etc/profile#讓配置文件立即生效
source /etc/profile

修改這幾個配置文件

這幾個配置文件的位置如下；
$HADOOP_HOME/etc/hadoop/

1.hadoop-env.sh

export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64

2.core-site.xml

<configuration><property><name>fs.defaultFS</name><value>hdfs://localhost:9000</value>  # 默認文件系統地址</property><property><name>hadoop.tmp.dir</name><value>/opt/hadoop/tmp</value>        # 臨時文件目錄:</property>
</configuration>

3.hdfs-site.xml

<property><name>dfs.replication</name><value>1</value>                        # 數據副本數（單機設置為1）
</property>
<property><name>dfs.namenode.name.dir</name><value>/opt/hadoop/hdfs/namenode</value>  # NameNode數據存儲路徑
</property>

4.mapred-site.xml

<property><name>mapreduce.framework.name</name><value>yarn</value>  # 啟用YARN框架
</property>

5.yarn-site.xml

<property><name>yarn.nodemanager.aux-services</name><value>mapreduce_shuffle</value>  # 指定NodeManager附加服務
</property>

格式化HDFS文件系統

hdfs namenode -format  # 初始化NameNode

啟動hadoop

1.啟動hdfs
start-dfs.sh2.啟動yarn
start-yarn.sh

查看集群啟動狀態及進程

jpshdfs dfsadmin -report

啟動hadoop報錯

./start-dfs.sh
Starting namenodes on [localhost]
ERROR: Attempting to operate on hdfs namenode as root
ERROR: but there is no HDFS_NAMENODE_USER defined. Aborting operation.
Starting datanodes
ERROR: Attempting to operate on hdfs datanode as root
ERROR: but there is no HDFS_DATANODE_USER defined. Aborting operation.
Starting secondary namenodes [localhost.localdomain]
ERROR: Attempting to operate on hdfs secondarynamenode as root
ERROR: but there is no HDFS_SECONDARYNAMENODE_USER defined. Aborting operation

解決方案，在配置文件中配置如下，即可用root用戶進行啟動了

四、hive的安裝

https://dlcdn.apache.org/hive/hive-4.0.1/

在MySQL中創建Hive元數據庫及用戶

CREATE DATABASE hive_meta;
CREATE USER 'hive'@'%' IDENTIFIED BY 'Jky1234!@#$';
GRANT ALL ON hive_meta.* TO 'hive'@'%';
FLUSH PRIVILEGES;

修改Hive配置文件

cd $HIVE_HOME/conf
cp hive-default.xml.template hive-site.xml
vi hive-site.xml  # 替換以下關鍵參數

hive的配置文件內容添加如下

<property><name>javax.jdo.option.ConnectionURL</name><value>jdbc:mysql://localhost:3306/hive_meta?createDatabaseIfNotExist=true</value>
</property>
<property><name>javax.jdo.option.ConnectionDriverName</name><value>com.mysql.cj.jdbc.Driver</value>
</property>
<property><name>javax.jdo.option.ConnectionUserName</name><value>hive</value>
</property>
<property><name>javax.jdo.option.ConnectionPassword</name><value>Jky1234!@#$</value>
</property>

下載mysql-connector-java-8.0.27.jar并拷貝到Hive的lib目錄

cp mysql-connector-java-8.0.27.jar /opt/hive/lib/opt/hadoop/share/hadoop/common/lib
cp guava-27.0-jre.jar /opt/hive/lib

在hive的bin目錄下執行初始化數據庫的操作

./schematool -dbType mysql -initSchema
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/opt/hive/lib/log4j-slf4j-impl-2.18.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/opt/hadoop/share/hadoop/common/lib/slf4j-reload4j-1.7.36.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
Initializing the schema to: 4.0.0
Metastore connection URL:	 jdbc:mysql://localhost:3306/hive_meta?createDatabaseIfNotExist=true
Metastore connection Driver :	 com.mysql.cj.jdbc.Driver
Metastore connection User:	 hive
Starting metastore schema initialization to 4.0.0
Initialization script hive-schema-4.0.0.mysql.sqlInitialization script completed

是否成功？去MySQL中

use hive_meta;
show tables;

啟動hive的客戶端，顯示如下；
仔細看日志打印的信息，應該是提示日志沖突了

./hive
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/opt/hive/lib/log4j-slf4j-impl-2.18.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/opt/hadoop/share/hadoop/common/lib/slf4j-reload4j-1.7.36.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/opt/hive/lib/log4j-slf4j-impl-2.18.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/opt/hadoop/share/hadoop/common/lib/slf4j-reload4j-1.7.36.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
Beeline version 4.0.1 by Apache Hive
beeline> show databases;
No current connection

ll log4j
-rw-r–r–. 1 root root 349845 9月 25 2024 log4j-1.2-api-2.18.0.jar
-rw-r–r–. 1 root root 315115 9月 25 2024 log4j-api-2.18.0.jar
-rw-r–r–. 1 root root 1861441 9月 25 2024 log4j-core-2.18.0.jar
-rw-r–r–. 1 root root 24801 9月 25 2024 log4j-slf4j-impl-2.18.0.jar
-rw-r–r–. 1 root root 36166 9月 25 2024 log4j-web-2.18.0.jar

mv log4j-slf4j-impl-2.18.0.jar log4j-slf4j-impl-2.18.0-bak.jar0526
然后在執行，發現多余的日志信息沒有了

執行連接hive的命令，報錯如下；

!connect jdbc:hive2://localhost:10000

./beeline --verbose
Default hs2 connection config file not found
Beeline version 4.0.1 by Apache Hive

查看1000的端口是否被監聽
ss -tulnp | grep 9870
tcp LISTEN 0 500 0.0.0.0:9870 0.0.0.0:* users:((“java”,pid=13987,fd=329))

./hive --service hiveserver2 --hiveconf hive.log.dir=/var/log/hive --hiveconf hive.log.file=hiveserver2-daemon.log
which: no hbase in (/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/usr/java/java-se-8u44-ri/bin:/opt/hadoop/bin:/opt/hadoop/sbin:/opt/hive/bin:/root/bin)
2025-05-26 18:16:09: Starting HiveServer2
Exception in thread “main” java.lang.IllegalArgumentException: Logs will be split in two files if the commandline argument hive.log.file is used. To prevent this use to HADOOP_CLIENT_OPTS -Dhive.log.file=hiveserver2-daemon.log or use the set the value in the configuration file (see HIVE-19886)
at org.apache.hive.service.server.HiveServer2$ServerOptionsProcessor.parse(HiveServer2.java:1366)
at org.apache.hive.service.server.HiveServer2.main(HiveServer2.java:1270)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at org.apache.hadoop.util.RunJar.run(RunJar.java:330)
at org.apache.hadoop.util.RunJar.main(RunJar.java:245)

./beeline --verbose
Overriding connection url property url_prefix from user connection configuration file
Connecting to jdbc:hive2://localhost:10000/default
Enter username for jdbc:hive2://localhost:10000/default: hive
Enter password for jdbc:hive2://localhost:10000/default: ****
25/05/26 18:23:26 [main]: WARN jdbc.HiveConnection: Failed to connect to localhost:10000
Error: Could not open client transport with JDBC Uri: jdbc:hive2://localhost:10000/default: Failed to open new session: java.lang.RuntimeException: org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.authorize.AuthorizationException): User: root is not allowed to impersonate hive (state=08S01,code=0)
java.sql.SQLException: Could not open client transport with JDBC Uri: jdbc:hive2://localhost:10000/default: Failed to open new session: java.lang.RuntimeException: org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.authorize.AuthorizationException): User: root is not allowed to impersonate hive
解決方案

hadoop.proxyuser.root.hosts

hadoop.proxyuser.root.groups