spark-sql讀寫同一張表，報錯Cannot overwrite a path that is also being read from

1. 增加checkpoint，設置檢查點阻斷血緣關系

sparkSession.sparkContext.setCheckpointDir("/tmp/spark/job/OrderOnlineSparkJob")val oldOneIdTagSql = s"select one_id,tag from aaa "
val oldOneIdTagDf = sparkSession.sql(oldOneIdTagSql).checkpoint()

2. 清理checkpoint產生的文件

2.1 更改配置文件開啟checkpoint文件清理

park.cleaner.referenceTracking.cleanCheckpoints = true

2.2 可以通過getCheckpointDir.get方式讀取到checkpoint的文件地址。從而自定義清理操作

val checkPointFile = sparkSession.sparkContext.getCheckpointDir.get
HdfsUtils.delete(fileSystem,checkPointFile,false)