Let me start by introducing two problems that I have dealt time and again with my experience with Apache Spark:

首先，我介紹一下我在Apache Spark上的經歷反復解決的兩個問題：

1. Data “overwrite” on the same path causing data loss in case of Job Failure.
   在作業失敗的情況下，同一路徑上的數據“覆蓋”會導致數據丟失。
2. Updates in the data.
   數據更新。

Sometimes I solved above with Design changes, sometimes with the introduction of another layer like Aerospike, or sometimes by maintaining historical incremental data.

有時我通過設計更改解決了上述問題，有時通過引入了諸如Aerospike的另一層，或者有時通過維護歷史增量數據來解決。

Maintaining historical data is mostly an immediate solution but I don’t really like dealing with historical incremental data if it’s not really required as(at least for me) it introduces the pain of backfill in case of failures which may be unlikely but inevitable.

維護歷史數據通常是一個立即解決方案，但是如果不是真正需要歷史增量數據，我真的不喜歡處理它，因為(至少對我來說)這會帶來回填的痛苦，以防止發生故障(雖然這不太可能，但不可避免)。

The above two problems are “problems” because Apache Spark does not really support ACID. I know it was never Spark’s use case to work with transactions(hello, you can’t have everything) but sometimes, there might be a scenario(like my two problems above) where ACID compliance would have come in handy.

以上兩個問題是“問題”，因為Apache Spark并不真正支持ACID。 我知道這絕不是Spark處理事務的用例(您好，您不能擁有所有東西)，但是有時候，在某些情況下(如上述兩個問題)，ACID合規性會派上用場。

When I read about Delta Lake and its ACID compliance, I saw it as one of the possible solutions for my two problems. Please read on to find out how the two problems are related to ACID compliance failure and how delta lake can be seen as a savior?

當我閱讀Delta Lake及其ACID合規性時，我將其視為解決我的兩個問題的可能解決方案之一。 請繼續閱讀，以找出這兩個問題與ACID合規性失敗之間的關系以及如何將三角洲湖視為救星？

什么是三角洲湖？ (What is Delta Lake?)

Delta Lake Documentation introduces Delta lake as:

Delta Lake文檔將Delta Lake引入為：

Delta Lake is an open source storage layer that brings reliability to data lakes. Delta Lake provides ACID transactions, scalable metadata handling, and unifies streaming and batch data processing. Delta Lake runs on top of your existing data lake and is fully compatible with Apache Spark APIs.

Delta Lake是一個開源存儲層 ，可為數據湖帶來可靠性。 Delta Lake提供ACID事務，可伸縮的元數據處理，并統一流和批處理數據處理。 Delta Lake在您現有的數據湖之上運行，并且與Apache Spark API完全兼容。

Delta Lake key points:

三角洲湖重點：

• Supports ACID
  支持ACID
• Enables Time travel
  啟用時間旅行
• Enables UPSERT
  啟用UPSERT

Spark如何使ACID失敗？ (How Spark fails ACID?)

Consider the following piece of code to remove duplicates from a dataset:

考慮以下代碼，從數據集中刪除重復項：

# Read from HDFS
df = spark.read.parquet("/path/on/hdfs") # Line 1
# Remove duplicates
df = df.distinct() # Line 2
# Overwrite the data
df.cache() # Line 3
df.write.parquet("/path/on/hdfs", mode="overwrite") # Line 4

For my spark application running above piece of code consider a scenario where it fails on Line 4, that is while writing the data. This may or may not lead to data loss. [Problem #1: As mentioned above].You can replicate the scenario, by creating a test dataset and kill the job when it’s in the Write stage.

對于在上述代碼段上運行的我的spark應用程序，請考慮一種情況，即它在第4行上失敗，即在寫入數據時。 這可能會或可能不會導致數據丟失。 [問題1：如上所述]。您可以通過創建測試數據集來復制方案，并在處于Write階段時取消該作業。

Let us try to understand ACID failure in spark with the above scenario.

讓我們嘗試了解上述情況下火花中的ACID故障。

ACID中的A代表原子性， (A in ACID stands for Atomicity,)

• What is Atomicity: Either all changes take place or none, the system is never in halfway state.

  什么是原子性：要么所有更改都發生，要么全部不發生，系統永遠不會處于中間狀態。

• How spark fails: While writing data, (at Line 4 above), if a failure occurs at a stage where old data is removed and new data is not yet written, data loss occurs. We have lost old data and we were not able to write new data due to job failure, atomicity fails. [It can vary according to file output committer used, please do read about File output committer to see how data writing takes place, the scenario I explained is for v2]

  火花如何失敗：在寫入數據時(在上面的第4行)，如果在刪除舊數據而尚未寫入新數據的階段發生故障，則會發生數據丟失。 我們丟失了舊數據，并且由于作業失敗，原子性失敗而無法寫入新數據。 [根據使用的文件輸出提交程序，它可能有所不同，請閱讀有關文件輸出提交程序的信息，以了解如何進行數據寫入，我所說明的場景適用于v2]

ACID中的C代表一致性， (C in ACID stands for Consistency,)

• What is Consistency: Data must be consistent and valid in the system at all times.

  什么是一致性 ：數據必須始終在系統中保持一致和有效。

• How Spark fails: As seen above, in the case of failure and data loss, we are left with invalid data in the system, consistency fails.

  Spark如何失敗：如上所示，在失敗和數據丟失的情況下，我們在系統中留有無效數據，一致性失敗。

ACID中的I代表隔離， (I in ACID stands for Isolation,)

• What is Isolation: Multiple transactions occur in isolation

  什么是隔離：多個事務是隔離發生的

• How spark fails: Consider two jobs running in parallel, one as described above and another which is also using the same dataset, if one job overwrites the dataset while other is still using it, failure might happen, isolation fails.

  Spark如何失敗：考慮兩個并行運行的作業，一個如上所述，另一個使用相同的數據集，如果一個作業覆蓋了數據集而另一個仍在使用它，則可能會發生故障，隔離失敗。

ACID中的D代表耐久性， (D in ACID stands for Durability,)

• What is Durability: Changes once made are never lost, even in the case of system failure.

  什么是耐久性：即使系統發生故障，一旦進行更改也不會丟失。

• How spark might fail: Spark really doesn’t affect the durability, it is mainly governed by the storage layer, but since we are losing data in case of job failures, in my opinion, it is a durability failure.

  Spark 可能如何失敗： Spark確實不會影響持久性，它主要由存儲層控制，但是由于我們在工作失敗的情況下會丟失數據，因此我認為這是持久性失敗。

Delta Lake如何支持ACID？ (How Delta Lake supports ACID?)

Delta lake maintains a delta log in the path where data is written. Delta Log maintains details like:

Delta Lake在寫入數據的路徑中維護一個delta日志。 Delta Log維護以下詳細信息：

• Metadata like

  像元數據

  - Paths added in the write operation.

  -在寫操作中添加的路徑。

  - Paths removed in the write operation.

  -在寫操作中刪除了路徑。

  - Data size

  -數據大小

  - Changes in data

  -數據變化

• Data Schema
  數據架構

• Commit information like

  提交信息，例如

  - Number of output rows

  -輸出行數

  - Output bytes

  -輸出字節

  - Timestamp

  -時間戳

Sample log file in _delta_log_ directory created after some operations:

某些操作后在_delta_log_目錄中創建的示例日志文件：

After successful execution, a log file is created in the _delta_log_ directory. The important thing to note is when you save your data as delta, no files once written are removed. The concept is similar to versioning.

成功執行后，將在_delta_log_目錄中創建一個日志文件。 需要注意的重要一點是，當您將數據另存為增量時，寫入后不會刪除任何文件。 該概念類似于版本控制。

By keeping track of paths removed, added and other metadata information in the _delta_log_, Delta lake is ACID-compliant.

通過跟蹤_delta_log_中路徑的刪除，添加和其他元數據信息，Delta Lake符合ACID。

Versioning enables time travel property of Delta Lake, which is, I can go back to any state of data because all this information is being maintained in _delta_log_.

版本控制啟用了Delta Lake的時間旅行屬性，即，由于所有這些信息都保存在_delta_log_中，因此我可以返回到任何數據狀態。

Delta Lake如何解決上述兩個問題？ (How Delta Lake solves my two problems mentioned above?)

• With the support for ACID, if my job fails during the “overwrite” operation, data is not lost, as changes won’t be committed to the log file of _delta_log_ directory. Also, since Delta Lake, does not remove old files in the “overwrite operation”, old state of my data is maintained and there is no data loss. (Yes, I have tested it)
  有了ACID的支持，如果我的工作在“覆蓋”操作期間失敗，則數據不會丟失，因為更改不會提交到_delta_log_目錄的日志文件中。 另外，由于Delta Lake不會在“覆蓋操作”中刪除舊文件，因此我的數據保持了舊狀態，并且沒有數據丟失。 (是的，我已經測試過了)
• Delta lake supports Update operation as mentioned above so it makes dealing with updates in data easier.
  Delta Lake支持如上所述的Update操作，因此使數據更新更容易。

Until next time,Ciao.

直到下次，Ciao。