前提：先安裝Kerbos
1、創建keytab目錄
在每臺機器上上提前創建好對應的kertab目錄
[hadoop@tv3-hadoop-01?~]$ sudo mkdir -p /BigData/run/hadoop/keytab/
[hadoop@tv3-hadoop-01?~]$ sudo mkdir -p /opt/security/
[hadoop@tv3-hadoop-01?~]$ sudo chown hadoop:hadoop /BigData/run/hadoop/keytab/
[hadoop@tv3-hadoop-01 ~]$ ls -lrt /BigData/run/hadoop/
drwxr-xr-x 2 hadoop hadoop? 4096 Jun 26 23:22 keytab
2、創建kerbos證書
進入管理機器，比如tv3-hadoop-01【本例中hadoop服務啟動統一使用hadoop用戶】
# 進入kadmin
[root@tv3-hadoop-01 ~]# kadmin.local
Authenticating as principal hadoop/admin@EXAMPLE.COM with password.
kadmin.local: ?
# 查看用戶
kadmin.local:? listprincs
# 創建用戶
addprinc -randkey hadoop/tv3-hadoop-01@EXAMPLE.COM
3、證書添加
依次增加其他hdfs節點的驗證，并導出到/BigData/run/hadoop/keytab/hadoop.keytab這個文件：
addprinc -randkey hadoop/tv3-hadoop-01@EXAMPLE.COM addprinc -randkey hadoop/tv3-hadoop-02@EXAMPLE.COM addprinc -randkey hadoop/tv3-hadoop-03@EXAMPLE.COM addprinc -randkey hadoop/tv3-hadoop-04@EXAMPLE.COM addprinc -randkey hadoop/tv3-hadoop-05@EXAMPLE.COM addprinc -randkey hadoop/tv3-hadoop-06@EXAMPLE.COMaddprinc -randkey HTTP/tv3-hadoop-01@EXAMPLE.COM addprinc -randkey HTTP/tv3-hadoop-02@EXAMPLE.COM addprinc -randkey HTTP/tv3-hadoop-03@EXAMPLE.COM addprinc -randkey HTTP/tv3-hadoop-04@EXAMPLE.COM addprinc -randkey HTTP/tv3-hadoop-05@EXAMPLE.COM addprinc -randkey HTTP/tv3-hadoop-06@EXAMPLE.COMktadd -k /BigData/run/hadoop/keytab/hadoop.keytab hadoop/tv3-hadoop-01@EXAMPLE.COM ktadd -k /BigData/run/hadoop/keytab/hadoop.keytab hadoop/tv3-hadoop-02@EXAMPLE.COM ktadd -k /BigData/run/hadoop/keytab/hadoop.keytab hadoop/tv3-hadoop-03@EXAMPLE.COM ktadd -k /BigData/run/hadoop/keytab/hadoop.keytab hadoop/tv3-hadoop-04@EXAMPLE.COM ktadd -k /BigData/run/hadoop/keytab/hadoop.keytab hadoop/tv3-hadoop-05@EXAMPLE.COM ktadd -k /BigData/run/hadoop/keytab/hadoop.keytab hadoop/tv3-hadoop-06@EXAMPLE.COMktadd -k /BigData/run/hadoop/keytab/HTTP.keytab HTTP/tv3-hadoop-01@EXAMPLE.COM ktadd -k /BigData/run/hadoop/keytab/HTTP.keytab HTTP/tv3-hadoop-02@EXAMPLE.COM ktadd -k /BigData/run/hadoop/keytab/HTTP.keytab HTTP/tv3-hadoop-03@EXAMPLE.COM ktadd -k /BigData/run/hadoop/keytab/HTTP.keytab HTTP/tv3-hadoop-04@EXAMPLE.COM ktadd -k /BigData/run/hadoop/keytab/HTTP.keytab HTTP/tv3-hadoop-05@EXAMPLE.COM ktadd -k /BigData/run/hadoop/keytab/HTTP.keytab HTTP/tv3-hadoop-06@EXAMPLE.COM
4、權限修改&kertab同步
修改權限到hadoop啟動用戶，否則會有權限訪問問題,并同步到其他hdfs所有服務的節點上（JN，DN，NN，RM，NM）
su - hadoopsudo chown hadoop:hadoop /BigData/run/hadoop/keytab/*.keytabscp /BigData/run/hadoop/keytab/hadoop.keytab /BigData/run/hadoop/keytab/HTTP.keytab hadoop@tv3-hadoop-06:/BigData/run/hadoop/keytab
5、修改配置文件
5.1 hdfs-site.xml
<property><name>dfs.block.access.token.enable</name><value>true</value><description>Enable HDFS block access tokens for secure operations</description></property><property><name>dfs.namenode.kerberos.principal</name><value>hadoop/_HOST@EXAMPLE.COM</value><description>namenode對應的kerberos賬戶為 nn/主機名@EXAMPLE.CPOM _HOST會自動轉換為主機名</description></property><property><name>dfs.namenode.keytab.file</name><value>/BigData/run/hadoop/keytab/hadoop.keytab</value><description>因為使用-randkey 創建的用戶 密碼隨機不知道，所以需要用免密登錄的keytab文件 指定namenode需要用的keytab文件在哪里</description></property><property><name>dfs.namenode.kerberos.internal.spnego.principal</name><value>HTTP/_HOST@EXAMPLE.COM</value><description>https 相關（如開啟namenodeUI）使用的賬戶</description></property><property><name>dfs.namenode.kerberos.internal.spnego.keytab</name><value>/BigData/run/hadoop/keytab/HTTP.keytab</value></property><property><name>dfs.secondary.namenode.kerberos.principal</name><value>hadoop/_HOST@EXAMPLE.COM</value><description>secondarynamenode使用的賬戶</description></property><property><name>dfs.secondary.namenode.keytab.file</name><value>/BigData/run/hadoop/keytab/hadoop.keytab</value><description>sn對應的keytab文件</description></property><property><name>dfs.secondary.namenode.kerberos.internal.spnego.principal</name><value>HTTP/_HOST@EXAMPLE.COM</value><description>sn需要開啟http頁面用到的賬戶</description></property><property><name>dfs.secondary.namenode.kerberos.internal.spnego.keytab</name><value>/BigData/run/hadoop/keytab/HTTP.keytab</value></property><property><name>dfs.journalnode.kerberos.principal</name><value>hadoop/_HOST@EXAMPLE.COM</value></property><property><name>dfs.journalnode.keytab.file</name><value>/BigData/run/hadoop/keytab/hadoop.keytab</value></property><property><name>dfs.journalnode.kerberos.internal.spnego.principal</name><value>HTTP/_HOST@EXAMPLE.COM</value></property><property><name>dfs.journalnode.kerberos.internal.spnego.keytab</name><value>/BigData/run/hadoop/keytab/HTTP.keytab</value></property><property><name>dfs.encrypt.data.transfer</name><value>true</value><description>數據傳輸協議激活數據加密</description></property><property><name>dfs.datanode.kerberos.principal</name><value>hadoop/_HOST@EXAMPLE.COM</value><description>datanode用到的賬戶</description></property><property><name>dfs.datanode.keytab.file</name><value>/BigData/run/hadoop/keytab/hadoop.keytab</value><description>datanode用到的keytab文件路徑</description></property><property><name>dfs.data.transfer.protection</name><value>integrity</value></property><property><name>dfs.https.port</name><value>50470</value></property><!-- required if hdfs support https --><property><name>dfs.http.policy</name><value>HTTPS_ONLY</value></property><!-- WebHDFS security config --><property><name>dfs.web.authentication.kerberos.principal</name><value>HTTP/_HOST@EXAMPLE.COM</value><description>web hdfs 使用的賬戶</description></property><property><name>dfs.web.authentication.kerberos.keytab</name><value>/BigData/run/hadoop/keytab/HTTP.keytab</value><description>對應的keytab文件</description></property>5.2 core-site.xml
<property><name>dfs.block.access.token.enable</name><value>true</value><description>Enable HDFS block access tokens for secure operations</description></property><property><name>hadoop.security.authorization</name><value>true</value><description>是否開啟hadoop的安全認證</description></property><property><name>hadoop.security.authentication</name><value>kerberos</value><description>使用kerberos作為hadoop的安全認證方案</description></property><property><name>hadoop.rpc.protection</name><value>authentication</value><description>authentication : authentication only (default); integrity : integrity check in addition to authentication; privacy : data encryption in addition to integrity</description></property><property><name>hadoop.security.auth_to_local</name><value>RULE:[2:$1@$0](hadoop@.*EXAMPLE.COM)s/.*/hadoop/RULE:[2:$1@$0](HTTP@.*EXAMPLE.COM)s/.*/hadoop/DEFAULT</value></property>5.3?yarn-site.xml
<property><name>hadoop.http.authentication.type</name><value>kerberos</value></property><property><name>hadoop.http.filter.initializers</name><value>org.apache.hadoop.security.AuthenticationFilterInitializer</value></property><property><name>yarn.resourcemanager.webapp.delegation-token-auth-filter.enabled</name><value>false</value><description>標記以啟用使用RM身份驗證篩選器覆蓋默認kerberos身份驗證篩選器以允許使用委派令牌進行身份驗證（如果缺少令牌，則回退到kerberos）。僅適用于http身份驗證類型為kerberos的情況。</description></property><property><name>hadoop.http.authentication.kerberos.principal</name><value>HTTP/_HOST@EXAMPLE.COM</value></property><property><name>hadoop.http.authentication.kerberos.keytab</name><value>/BigData/run/hadoop/keytab/HTTP.keytab</value></property><property><name>yarn.acl.enable</name><value>true</value></property><property><name>yarn.web-proxy.principal</name><value>HTTP/_HOST@EXAMPLE.COM</value></property><property><name>yarn.web-proxy.keytab</name><value>/BigData/run/hadoop/keytab/HTTP.keytab</value></property><property><name>yarn.resourcemanager.principal</name><value>hadoop/_HOST@EXAMPLE.COM</value></property><property><name>yarn.resourcemanager.keytab</name><value>/BigData/run/hadoop/keytab/hadoop.keytab</value></property><!-- nodemanager --><property><name>yarn.nodemanager.principal</name><value>hadoop/_HOST@EXAMPLE.COM</value></property><property><name>yarn.nodemanager.keytab</name><value>/BigData/run/hadoop/keytab/hadoop.keytab</value></property><property><name>yarn.nodemanager.container-executor.class</name><value>org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor</value>
</property><property><name>yarn.nodemanager.linux-container-executor.group</name><value>hadoop</value></property><property><name>yarn.nodemanager.linux-container-executor.path</name><value>/BigData/run/hadoop/bin/container-executor</value></property><!-- webapp webapp configs --><property><name>yarn.resourcemanager.webapp.spnego-principal</name><value>HTTP/_HOST@EXAMPLE.COM</value></property><property><name>yarn.resourcemanager.webapp.spnego-keytab-file</name><value>/BigData/run/hadoop/keytab/HTTP.keytab</value></property><property><name>yarn.timeline-service.http-authentication.type</name><value>kerberos</value><description>Defines authentication used for the timeline server HTTP endpoint. Supported values are: simple | kerberos | #AUTHENTICATION_HANDLER_CLASSNAME#</description></property><property><name>yarn.timeline-service.principal</name><value>hadoop/_HOST@EXAMPLE.COM</value></property><property><name>yarn.timeline-service.keytab</name><value>/BigData/run/hadoop/keytab/hadoop.keytab</value></property><property><name>yarn.timeline-service.http-authentication.kerberos.principal</name><value>HTTP/_HOST@EXAMPLE.COM</value></property><property> <name>yarn.timeline-service.http-authentication.kerberos.keytab</name><value>/BigData/run/hadoop/keytab/HTTP.keytab</value></property><property><name>yarn.nodemanager.container-localizer.java.opts</name><value>-Djava.security.krb5.conf=/etc/krb5.conf -Djava.security.krb5.realm=EXAMPLE.COM -Djava.security.krb5.kdc=tv3-hadoop-01:88</value></property><property><name>yarn.nodemanager.health-checker.script.opts</name><value>-Djava.security.krb5.conf=/etc/krb5.conf -Djava.security.krb5.realm=EXAMPLE.COM -Djava.security.krb5.kdc=tv3-hadoop-01:88</value></property><property><name>yarn.nodemanager.linux-container-executor.nonsecure-mode.local-user</name><value>hadoop</value></property><property><name>yarn.nodemanager.linux-container-executor.group</name><value>hadoop</value></property>5.4?mapred-site.xml
<property><name>mapreduce.map.java.opts</name><value>-Xmx1638M -Djava.security.krb5.conf=/etc/krb5.conf -Djava.security.krb5.realm=EXAMPLE.COM -Djava.security.krb5.kdc=tv3-hadoop-01:88</value>
</property><property><name>mapreduce.reduce.java.opts</name><value>-Xmx3276M -Djava.security.krb5.conf=/etc/krb5.conf -Djava.security.krb5.realm=EXAMPLE.COM -Djava.security.krb5.kdc=tv3-hadoop-01:88</value>
</property><property><name>mapreduce.jobhistory.keytab</name><value>/BigData/run/hadoop/keytab/hadoop.keytab</value>
</property><property><name>mapreduce.jobhistory.principal</name><value>hadoop/_HOST@EXAMPLE.COM</value>
</property><property><name>mapreduce.jobhistory.webapp.spnego-keytab-file</name><value>/BigData/run/hadoop/keytab/HTTP.keytab</value>
</property><property><name>mapreduce.jobhistory.webapp.spnego-principal</name><value>HTTP/_HOST@EXAMPLE.COM</value>
</property>5.5 配置文件同步到各個節點
cd /BigData/run/hadoop/etc/hadoop
scp hdfs-site.xml yarn-site.xml core-site.xml mapred-site.xml hadoop@tv3-hadoop-06:/BigData/run/hadoop/etc/hadoop/
6、配置SSL（開啟https）
6.1 創建https證書(需要在每臺機器上執行)
?
[hadoop@tv3-hadoop-01?hadoop]# mkdir -p /opt/security/kerberos_https
[hadoop@tv3-hadoop-01 hadoop]# cd /opt/security/kerberos_https
6.2 在任意一個hadoop節點生成CA證書
[root@tv3-hadoop-01 kerberos_https]# openssl req -new -x509 -keyout hdfs_ca_key -out hdfs_ca_cert -days 9999 -subj /C=CN/ST=shanxi/L=xian/O=hlk/OU=hlk/CN=tv3-hadoop01
Generating a 2048 bit RSA private key
...........................................................................................+++
.................................................................................+++
writing new private key to 'hdfs_ca_key'
Enter PEM pass phrase:
Verifying - Enter PEM pass phrase:
-----
[root@tv3-hadoop-01 kerberos_https]# ls -lrt
total 8
-rw-r--r-- 1 root root 1834 Jun 29 09:45 hdfs_ca_key
-rw-r--r-- 1 root root 1302 Jun 29 09:45 hdfs_ca_cert6.3 將上面生成的CA 證書發送到每個節點上
scp -r /opt/security/kerberos_https root@tv3-hadoop-06:/opt/security/?6.4 在每個hadoop節點上制作證書
cd /opt/security/kerberos_https# 所有需要輸入密碼的地方全部輸入123456（方便起見，如果你對密碼有要求請自行修改）# 1 輸入密碼和確認密碼：123456，此命令成功后輸出keystore文件name="CN=$HOSTNAME, OU=hlk, O=hlk, L=xian, ST=shanxi, C=CN"#需要輸入第一步輸入的密碼四次keytool -keystore keystore -alias localhost -validity 9999 -genkey -keyalg RSA -keysize 2048 -dname "$name"# 2 輸入密碼和確認密碼：123456，提示是否信任證書：輸入yes，此命令成功后輸出truststore文件keytool -keystore truststore -alias CARoot -import -file hdfs_ca_cert# 3 輸入密碼和確認密碼：123456，此命令成功后輸出cert文件keytool -certreq -alias localhost -keystore keystore -file cert# 4 此命令成功后輸出cert_signed文件openssl x509 -req -CA hdfs_ca_cert -CAkey hdfs_ca_key -in cert -out cert_signed -days 9999 -CAcreateserial# 5 輸入密碼和確認密碼：123456，是否信任證書，輸入yes，此命令成功后更新keystore文件keytool -keystore keystore -alias CARoot -import -file hdfs_ca_certkeytool -keystore keystore -alias localhost -import -file cert_signed[root@tv3-hadoop-06 kerberos_https]# ls -lrt
total 28
-rw-r--r-- 1 root root 1302 Jun 29 09:57 hdfs_ca_cert
-rw-r--r-- 1 root root 1834 Jun 29 09:57 hdfs_ca_key
-rw-r--r-- 1 root root 984 Jun 29 10:03 truststore
-rw-r--r-- 1 root root 1085 Jun 29 10:03 cert
-rw-r--r-- 1 root root 17 Jun 29 10:04 hdfs_ca_cert.srl
-rw-r--r-- 1 root root 1188 Jun 29 10:04 cert_signed
-rw-r--r-- 1 root root 4074 Jun 29 10:04 keystore6.5?修改SSL server文件
在${HADOOP_HOME}/etc/hadoop目錄構建ssl-server.xml文件
<configuration><property><name>ssl.server.truststore.location</name><value>/opt/security/kerberos_https/truststore</value><description>Truststore to be used by NN and DN. Must be specified.</description></property><property><name>ssl.server.truststore.password</name><value>123456</value><description>Optional. Default value is "". </description></property><property><name>ssl.server.truststore.type</name><value>jks</value><description>Optional. The keystore file format, default value is "jks".</description></property><property><name>ssl.server.truststore.reload.interval</name><value>10000</value><description>Truststore reload check interval, in milliseconds. Default value is 10000 (10 seconds). </description></property><property><name>ssl.server.keystore.location</name><value>/opt/security/kerberos_https/keystore</value><description>Keystore to be used by NN and DN. Must be specified.</description></property><property><name>ssl.server.keystore.password</name><value>123456</value><description>Must be specified.</description></property><property><name>ssl.server.keystore.keypassword</name><value>123456</value><description>Must be specified.</description></property><property><name>ssl.server.keystore.type</name><value>jks</value><description>Optional. The keystore file format, default value is "jks".</description></property><property><name>ssl.server.exclude.cipher.list</name><value>TLS_ECDHE_RSA_WITH_RC4_128_SHA,SSL_DHE_RSA_EXPORT_WITH_DES40_CBC_SHA,SSL_RSA_WITH_DES_CBC_SHA,SSL_DHE_RSA_WITH_DES_CBC_SHA,SSL_RSA_EXPORT_WITH_RC4_40_MD5,SSL_RSA_EXPORT_WITH_DES40_CBC_SHA,SSL_RSA_WITH_RC4_128_MD5</value><description>Optional. The weak security cipher suites that you want excludedfrom SSL communication.</description></property></configuration>6.6 修改SSL-client文件
<configuration><property><name>ssl.client.truststore.location</name><value>/opt/security/kerberos_https/truststore</value><description>Truststore to be used by clients like distcp. Must be specified. </description></property><property><name>ssl.client.truststore.password</name><value>123456</value><description>Optional. Default value is "". </description></property><property><name>ssl.client.truststore.type</name><value>jks</value><description>Optional. The keystore file format, default value is "jks".</description></property><property><name>ssl.client.truststore.reload.interval</name><value>10000</value><description>Truststore reload check interval, in milliseconds. Default value is 10000 (10 seconds). </description></property><property><name>ssl.client.keystore.location</name><value>/opt/security/kerberos_https/keystore</value><description>Keystore to be used by clients like distcp. Must be specified. </description></property><property><name>ssl.client.keystore.password</name><value>123456</value><description>Optional. Default value is "". </description></property><property><name>ssl.client.keystore.keypassword</name><value>123456</value><description>Optional. Default value is "". </description></property><property><name>ssl.client.keystore.type</name><value>jks</value><description>Optional. The keystore file format, default value is "jks". </description></property></configuration>
6.7 hdfs配置HTTPS（修改后需要同步到每個節點）
? <property><name>dfs.http.policy</name><value>HTTPS_ONLY</value><description>所有開啟的web頁面均使用https, 細節在ssl server 和client那個配置文件內配置</description></property>7、啟動hadoop基礎測試
7.1 HA模式啟動順序
建議依次啟動JN、NN、ZKFC、DN、RM、NM服務
7.2?啟動JN(每個服務啟動之前需要init單獨的節點）
kinit -kt /BigData/run/hadoop/keytab/hadoop.keytab hadoop/$HOSTNAME@EXAMPLE.COM
##重啟JournalNodehadoop-daemon.sh stop journalnode && hadoop-daemon.sh start journalnode##啟動JournalNodehadoop-daemon.sh start journalnode##停止JournalNodehadoop-daemon.sh stop journalnode7.3?啟動NameNode和ZKFC服務
如果是新集群，需要提前format
hadoop namenode -formatkinit -kt /BigData/run/hadoop/keytab/hadoop.keytab hadoop/$HOSTNAME@EXAMPLE.COM
##重啟nnhadoop-daemon.sh stop namenode && hadoop-daemon.sh start namenode##啟動nnhadoop-daemon.sh start namenode##停止nnhadoop-daemon.sh stop namenode##重啟zkfchadoop-daemon.sh stop zkfc && hadoop-daemon.sh start zkfc##啟動zkfchadoop-daemon.sh start zkfc##停止zkfchadoop-daemon.sh stop zkfc
7.4?啟動DatanNode服務
kinit -kt /BigData/run/hadoop/keytab/hadoop.keytab hadoop/$HOSTNAME@EXAMPLE.COM
##重啟dnhadoop-daemon.sh stop datanode && hadoop-daemon.sh start datanode##啟動dnhadoop-daemon.sh start datanode##停止dnhadoop-daemon.sh stop datanode
7.5?驗證HA功能(多NameNode)
[hadoop@tv3-hadoop-01 hadoop]$ hdfs haadmin -failover nn2 nn17.6?驗證HDFS文件讀寫
[hadoop@tv3-hadoop-01 ~]$ echo '123' > b
[hadoop@tv3-hadoop-01 ~]$ hdfs dfs -put -f b /tmp/
[hadoop@tv3-hadoop-01 ~]$ hdfs dfs -cat /tmp/b
123
[hadoop@tv3-hadoop-01 ~]$ 7.7?啟動HTTPS后 webui無法訪問UI狀態
7.8?啟動Resoucemanager服務
kinit -kt /BigData/run/hadoop/keytab/hadoop.keytab hadoop/$HOSTNAME@EXAMPLE.COM
##重啟rmyarn --daemon stop resourcemanager && yarn --daemon start resourcemanager##啟動rmyarn --daemon start resourcemanager##停止rmyarn --daemon stop resourcemanager7.9. 啟動Nodemanager服務
kinit -kt /BigData/run/hadoop/keytab/hadoop.keytab hadoop/$HOSTNAME@EXAMPLE.COM
##重啟rmyarn --daemon stop nodemanager && yarn --daemon start nodemanager
##啟動rmyarn --daemon start nodemanager##停止rmyarn --daemon stop nodemanager7.10?驗證Mapreduce job
hadoop jar /BigData/run/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-examples-3.3.1.jar pi 5 10
看到下面結果代表YARN已經部署ok
Job Finished in 66.573 seconds
Estimated value of Pi is 3.28000000000000000000
[hadoop@tv3-hadoop-01 hadoop]$ )
)
