個人博客地址:Tez 0.10.1安裝 | 一張假鈔的真實世界
具體安裝步驟參照官網安裝手冊即可。此處只對官網手冊進行補充。
從官網下載apache-tez-0.10.1-bin.tar.gz進行安裝未成功,出現下面的異常。最終按照官網源代碼編譯的方式安裝測試成功。
環境
- Hadoop 3.2.2
- Tez 0.10.1
二進制包安裝異常
- 異常1:
2021-08-18 18:51:15,362 [INFO] [main] |service.AbstractService|: Service org.apache.tez.dag.app.DAGAppMaster failed in state STARTED
org.apache.hadoop.service.ServiceStateException: java.lang.NoClassDefFoundError: org/eclipse/jetty/servlet/ServletContextHandlerat org.apache.hadoop.service.ServiceStateException.convert(ServiceStateException.java:105)at org.apache.tez.dag.app.DAGAppMaster.startServices(DAGAppMaster.java:1879)at org.apache.tez.dag.app.DAGAppMaster.serviceStart(DAGAppMaster.java:1943)at org.apache.hadoop.service.AbstractService.start(AbstractService.java:194)at org.apache.tez.dag.app.DAGAppMaster$9.run(DAGAppMaster.java:2626)at java.security.AccessController.doPrivileged(Native Method)at javax.security.auth.Subject.doAs(Subject.java:422)at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1762)at org.apache.tez.dag.app.DAGAppMaster.initAndStartAppMaster(DAGAppMaster.java:2622)at org.apache.tez.dag.app.DAGAppMaster.main(DAGAppMaster.java:2409)
Caused by: java.lang.NoClassDefFoundError: org/eclipse/jetty/servlet/ServletContextHandlerat org.apache.hadoop.yarn.webapp.WebApps.$for(WebApps.java:515)at org.apache.hadoop.yarn.webapp.WebApps.$for(WebApps.java:521)at org.apache.hadoop.yarn.webapp.WebApps.$for(WebApps.java:526)at org.apache.tez.dag.app.web.WebUIService.serviceStart(WebUIService.java:92)at org.apache.hadoop.service.AbstractService.start(AbstractService.java:194)at org.apache.tez.dag.app.DAGAppMaster$ServiceWithDependency.start(DAGAppMaster.java:1800)at org.apache.tez.dag.app.DAGAppMaster$ServiceThread.run(DAGAppMaster.java:1821)
Caused by: java.lang.ClassNotFoundException: org.eclipse.jetty.servlet.ServletContextHandlerat java.net.URLClassLoader.findClass(URLClassLoader.java:381)at java.lang.ClassLoader.loadClass(ClassLoader.java:424)at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349)at java.lang.ClassLoader.loadClass(ClassLoader.java:357)... 7 more
- 異常2:
2021-08-18 16:18:36,244 [INFO] [main] |service.AbstractService|: Service org.apache.tez.dag.app.DAGAppMaster failed in state STARTED
org.apache.hadoop.service.ServiceStateException: java.lang.NoSuchMethodError: org.eclipse.jetty.server.session.SessionHandler.getSessionManager()Lorg/eclipse/jetty/server/SessionManager;at org.apache.hadoop.service.ServiceStateException.convert(ServiceStateException.java:105)at org.apache.tez.dag.app.DAGAppMaster.startServices(DAGAppMaster.java:1879)at org.apache.tez.dag.app.DAGAppMaster.serviceStart(DAGAppMaster.java:1943)at org.apache.hadoop.service.AbstractService.start(AbstractService.java:194)at org.apache.tez.dag.app.DAGAppMaster$9.run(DAGAppMaster.java:2626)at java.security.AccessController.doPrivileged(Native Method)at javax.security.auth.Subject.doAs(Subject.java:422)at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729)at org.apache.tez.dag.app.DAGAppMaster.initAndStartAppMaster(DAGAppMaster.java:2622)at org.apache.tez.dag.app.DAGAppMaster.main(DAGAppMaster.java:2409)
Caused by: java.lang.NoSuchMethodError: org.eclipse.jetty.server.session.SessionHandler.getSessionManager()Lorg/eclipse/jetty/server/SessionManager;at org.apache.hadoop.http.HttpServer2.initializeWebServer(HttpServer2.java:569)at org.apache.hadoop.http.HttpServer2.<init>(HttpServer2.java:550)at org.apache.hadoop.http.HttpServer2.<init>(HttpServer2.java:117)at org.apache.hadoop.http.HttpServer2$Builder.build(HttpServer2.java:425)at org.apache.hadoop.yarn.webapp.WebApps$Builder.build(WebApps.java:341)at org.apache.hadoop.yarn.webapp.WebApps$Builder.start(WebApps.java:432)at org.apache.hadoop.yarn.webapp.WebApps$Builder.start(WebApps.java:428)at org.apache.tez.dag.app.web.WebUIService.serviceStart(WebUIService.java:94)at org.apache.hadoop.service.AbstractService.start(AbstractService.java:194)at org.apache.tez.dag.app.DAGAppMaster$ServiceWithDependency.start(DAGAppMaster.java:1800)at org.apache.tez.dag.app.DAGAppMaster$ServiceThread.run(DAGAppMaster.java:1821)
安裝說明
pom.xml
修改以下配置與使用的Hadoop版本一致:
<hadoop.version>3.2.2</hadoop.version>
修改以下配置與${HADOOP_HOME}/share/hadoop/common/lib/guava-27.0-jre.jar
版本一致。
<guava.version>27.0-jre</guava.version>
tez-site.xml
需要注意兩點:
- 與官網不同,
tez.use.cluster.hadoop-libs
配置為true。 - 該配置文件需要分發到Hadoop集群的所有節點。位置:
${HADOOP_HOME}/etc/hadoop
目錄下。
完整內容如下:
<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<configuration><property><name>tez.lib.uris</name><value>${fs.defaultFS}/lib/tez/tez-0.10.1.tar.gz</value><type>string</type></property><property><name>tez.use.cluster.hadoop-libs</name><value>true</value></property>
</configuration>