Error message from spark is:java.lang.Exception: application_1678793738534_17900289 Driver Disassociated [akka.tcp://sparkDriverClient@11.71.243.117:37931] <- [akka.tcp://sparkYarnSQLAM@9.10.130.149:38513] disassociated!

?

日志里頻繁full gc ，可以用這里參數替換重跑看下：
set hive.exec.parallel = true;set hive.execute.engine=spark;set hive.multi.join.use.hive=false;set hive.spark.failed.retry=false;set spark.sql.relation.ignoreDefaultPartition=false;set spark.network.timeout=1200;set spark.driver.memory=16g; set spark.executor.memory=16g;set spark.partition.max=15999; set spark.task.shuffle.size = 536870910;set spark.sql.broadcastTimeout = 36000;