azkaban運行任務的時候失敗報錯如下：

23-03-2016 08:16:14 CST analyzer-kafka2hdfs_new ERROR - Exception in thread "main" org.apache.hive.service.cli.HiveSQLException: java.lang.RuntimeException: The root scratch dir: /tmp/hive on HDFS should be writable. Current permissions are: rwxr-xr-x

23-03-2016 08:16:14 CST analyzer-kafka2hdfs_new ERROR - ?? ?at org.apache.hive.jdbc.Utils.verifySuccess(Utils.java:231)

23-03-2016 08:16:14 CST analyzer-kafka2hdfs_new ERROR - ?? ?at org.apache.hive.jdbc.Utils.verifySuccess(Utils.java:222)

23-03-2016 08:16:14 CST analyzer-kafka2hdfs_new ERROR - ?? ?at org.apache.hive.jdbc.HiveConnection.openSession(HiveConnection.java:459)

23-03-2016 08:16:14 CST analyzer-kafka2hdfs_new ERROR - ?? ?at org.apache.hive.jdbc.HiveConnection.(HiveConnection.java:178)

23-03-2016 08:16:14 CST analyzer-kafka2hdfs_new ERROR - ?? ?at org.apache.hive.jdbc.HiveDriver.connect(HiveDriver.java:105)

23-03-2016 08:16:14 CST analyzer-kafka2hdfs_new ERROR - ?? ?at java.sql.DriverManager.getConnection(DriverManager.java:571)

23-03-2016 08:16:14 CST analyzer-kafka2hdfs_new ERROR - ?? ?at java.sql.DriverManager.getConnection(DriverManager.java:215)

23-03-2016 08:16:14 CST analyzer-kafka2hdfs_new ERROR - ?? ?at com.geo.gdata.common.HiveClientUtils.getConnection(HiveClientUtils.java:36)

23-03-2016 08:16:14 CST analyzer-kafka2hdfs_new ERROR - ?? ?at com.geo.gdata.hive.DataToHive.loadDataToTab(DataToHive.java:59)

23-03-2016 08:16:14 CST analyzer-kafka2hdfs_new ERROR - ?? ?at com.geo.gdata.chain.KafkaToHiveTab.main(KafkaToHiveTab.java:74)

23-03-2016 08:16:14 CST analyzer-kafka2hdfs_new ERROR - ?? ?at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

23-03-2016 08:16:14 CST analyzer-kafka2hdfs_new ERROR - ?? ?at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)

23-03-2016 08:16:14 CST analyzer-kafka2hdfs_new ERROR - ?? ?at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

23-03-2016 08:16:14 CST analyzer-kafka2hdfs_new ERROR - ?? ?at java.lang.reflect.Method.invoke(Method.java:606)

23-03-2016 08:16:14 CST analyzer-kafka2hdfs_new ERROR - ?? ?at org.apache.hadoop.util.RunJar.main(RunJar.java:212)

23-03-2016 08:16:14 CST analyzer-kafka2hdfs_new ERROR - Caused by: java.lang.RuntimeException: java.lang.RuntimeException: The root scratch dir: /tmp/hive on HDFS should be writable. Current permissions are: rwxr-xr-x

23-03-2016 08:16:14 CST analyzer-kafka2hdfs_new ERROR - ?? ?at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:444)

23-03-2016 08:16:14 CST analyzer-kafka2hdfs_new ERROR - ?? ?at org.apache.hive.service.cli.session.HiveSessionImpl.(HiveSessionImpl.java:116)

23-03-2016 08:16:14 CST analyzer-kafka2hdfs_new ERROR - ?? ?at org.apache.hive.service.cli.session.HiveSessionImplwithUGI.(HiveSessionImplwithUGI.java:47)

23-03-2016 08:16:14 CST analyzer-kafka2hdfs_new ERROR - ?? ?at org.apache.hive.service.cli.session.SessionManager.openSession(SessionManager.java:260)

23-03-2016 08:16:14 CST analyzer-kafka2hdfs_new ERROR - ?? ?at org.apache.hive.service.cli.CLIService.openSessionWithImpersonation(CLIService.java:175)

23-03-2016 08:16:14 CST analyzer-kafka2hdfs_new ERROR - ?? ?at org.apache.hive.service.cli.thrift.ThriftCLIService.getSessionHandle(ThriftCLIService.java:322)

23-03-2016 08:16:14 CST analyzer-kafka2hdfs_new ERROR - ?? ?at org.apache.hive.service.cli.thrift.ThriftCLIService.OpenSession(ThriftCLIService.java:235)

23-03-2016 08:16:14 CST analyzer-kafka2hdfs_new ERROR - ?? ?at org.apache.hive.service.cli.thrift.TCLIService$Processor$OpenSession.getResult(TCLIService.java:1253)

23-03-2016 08:16:14 CST analyzer-kafka2hdfs_new ERROR - ?? ?at org.apache.hive.service.cli.thrift.TCLIService$Processor$OpenSession.getResult(TCLIService.java:1238)

23-03-2016 08:16:14 CST analyzer-kafka2hdfs_new ERROR - ?? ?at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)

23-03-2016 08:16:14 CST analyzer-kafka2hdfs_new ERROR - ?? ?at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)

23-03-2016 08:16:14 CST analyzer-kafka2hdfs_new ERROR - ?? ?at org.apache.hive.service.auth.TSetIpAddressProcessor.process(TSetIpAddressProcessor.java:56)

23-03-2016 08:16:14 CST analyzer-kafka2hdfs_new ERROR - ?? ?at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:206)

23-03-2016 08:16:14 CST analyzer-kafka2hdfs_new ERROR - ?? ?at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)

23-03-2016 08:16:14 CST analyzer-kafka2hdfs_new ERROR - ?? ?at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)

23-03-2016 08:16:14 CST analyzer-kafka2hdfs_new ERROR - ?? ?at java.lang.Thread.run(Thread.java:745)

23-03-2016 08:16:14 CST analyzer-kafka2hdfs_new ERROR - Caused by: java.lang.RuntimeException: The root scratch dir: /tmp/hive on HDFS should be writable. Current permissions are: rwxr-xr-x

23-03-2016 08:16:14 CST analyzer-kafka2hdfs_new ERROR - ?? ?at org.apache.hadoop.hive.ql.session.SessionState.createRootHDFSDir(SessionState.java:529)

23-03-2016 08:16:14 CST analyzer-kafka2hdfs_new ERROR - ?? ?at org.apache.hadoop.hive.ql.session.SessionState.createSessionDirs(SessionState.java:478)

23-03-2016 08:16:14 CST analyzer-kafka2hdfs_new ERROR - ?? ?at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:430)

23-03-2016 08:16:14 CST analyzer-kafka2hdfs_new ERROR - ?? ?... 15 more

23-03-2016 08:16:15 CST analyzer-kafka2hdfs_new INFO - Process completed unsuccessfully in 73 seconds.

23-03-2016 08:16:15 CST analyzer-kafka2hdfs_new ERROR - Job run failed!

23-03-2016 08:16:15 CST analyzer-kafka2hdfs_new ERROR - azkaban.jobExecutor.utils.process.ProcessFailureExceptionazkaban.jobExecutor.utils.process.ProcessFailureException

23-03-2016 08:16:15 CST analyzer-kafka2hdfs_new INFO - Finishing job analyzer-kafka2hdfs_new at 1458692175992 with status FAILED

問題分析：

從報錯中看到應該是跟權限有關系，搜索了相關文檔，基本可以肯定是權限問題。

解決方法：

將/tmp/hive置為777權限

Update the permission of your /tmp/hive HDFS directory using the following command

hadoop fs -chmod 777 /tmp/hive

hdfs dfs-chmod 777 /tmp/hive

If so can you remove /tmp/hive on both local and hdfs.

hadoop fs -rm -r /tmp/hive;

rm -rf /tmp/hive

Only temporary files are kept in this location. No problem even if we delete this, will be created when required with proper permissions.

閱讀(8275) | 評論(0) | 轉發(0) |