背景

本文基于Spark 4.0

總結

Spark中的 VariantType 類型，用盡量少的字節來存儲Json的格式化數據

分析

這里主要介紹 Variant 的存儲，我們從VariantBuilder.buildJson方法(把對應的json數據存儲為VariantType類型)開始：

public static Variant parseJson(JsonParser parser, boolean allowDuplicateKeys)throws IOException {VariantBuilder builder = new VariantBuilder(allowDuplicateKeys);builder.buildJson(parser);return builder.result();}

這個方法會調用buildJson這個方法：

JsonToken token = parser.currentToken();if (token == null) {throw new JsonParseException(parser, "Unexpected null token");}switch (token) {case START_OBJECT: {ArrayList<FieldEntry> fields = new ArrayList<>();int start = writePos;while (parser.nextToken() != JsonToken.END_OBJECT) {String key = parser.currentName();parser.nextToken();int id = addKey(key);fields.add(new FieldEntry(key, id, writePos - start));buildJson(parser);}finishWritingObject(start, fields);break;}case START_ARRAY: {ArrayList<Integer> offsets = new ArrayList<>();int start = writePos;while (parser.nextToken() != JsonToken.END_ARRAY) {offsets.add(writePos - start);buildJson(parser);}finishWritingArray(start, offsets);break;}case VALUE_STRING:appendString(parser.getText());break;case VALUE_NUMBER_INT:try {appendLong(parser.getLongValue(