Skywalking-9.6.0系列之本地源碼編譯并啟動

Skywalking相信有很多人使用過，通過容器或者下載安裝包進行安裝的，今天從源代碼角度，拉取、構建、啟動。
官方文檔步驟簡潔明了，我這邊會結合自己遇到的一些問題做出總結。

當前構建資源版本：

MAC 10.15.7
IDEA 2021.2
node v16.14.0 (據代碼要求安裝了一個新版本)
npm 8.3.1
Skywalking 9.6.0-SNAPSHOT
JDK 11 (最新的代碼要看最新的要求，注意不能使用JDK8，用錯的話構建時也會有提示)
Maven 3.6.0

官方指南(必看)：https://github.com/apache/skywalking/blob/master/docs/en/guides/How-to-build.md

快速閱覽

流程1：源碼獲取
流程2：submodule模塊初始化
或者流程1&2：二合一
流程3：構建項目
流程4：構建產生的源代碼數據加入項目索引
流程5：啟動前后端代碼

流程1：源碼獲取

方法一：fork代碼后，在自己的倉庫下 git clone源代碼
方法二：下載源碼包后解壓

【下載失敗，網絡問題，只能自行解決】

流程2：submodule模塊初始化

.gitsubmodules文件

[submodule "apm-protocol/apm-network/src/main/proto"]path = apm-protocol/apm-network/src/main/protourl = https://github.com/apache/skywalking-data-collect-protocol.git
[submodule "oap-server/server-query-plugin/query-graphql-plugin/src/main/resources/query-protocol"]path = oap-server/server-query-plugin/query-graphql-plugin/src/main/resources/query-protocolurl = https://github.com/apache/skywalking-query-protocol.git
[submodule "test/e2e-v2/java-test-service/e2e-protocol/src/main/proto"]path = test/e2e-v2/java-test-service/e2e-protocol/src/main/protourl = https://github.com/apache/skywalking-data-collect-protocol.git
[submodule "skywalking-ui"]path = skywalking-uiurl = https://github.com/apache/skywalking-booster-ui.git

以上可以看出關聯的倉庫數據主要有：

https://github.com/apache/skywalking-data-collect-protocol.git
https://github.com/apache/skywalking-query-protocol.git
https://github.com/apache/skywalking-data-collect-protocol.git
https://github.com/apache/skywalking-booster-ui.git

執行以下命令，來獲取這些代碼數據：

git submodule init //初始化倉庫目錄位置
git submodule update //**這步驟至關重要**

這個環節由于涉及四個倉庫的代碼拉取，此時如果網絡狀況不佳，很有可能存在【空文件夾】的情況，所以可以這樣解決：

方法一：保障網絡穩定，刪除拉取的空文件夾，再重新git submodule init AND git submodule update ，重復嘗試直至拉取成功
關注這些項目文件夾:

apm-protocol/apm-network/src/main/proto
oap-server/server-query-plugin/query-graphql-plugin/src/main/resources/query-protocol
test/e2e-v2/java-test-service/e2e-protocol/src/main/proto
skywalking-ui

方法二：去除.gitsubmodules文件中的內容，按照需求取拉取對應的代碼，參照【gitsubmodules】文件目錄要求放到指定的文件夾內

以上，可能存在的問題主要就是網絡方面，如果以上項目目錄未拉全就去執行下一步構建，會產生很多報錯，這步必須確認好

或者流程1&2：二合一

git clone --recurse-submodules https://github.com/apache/skywalking.git

也就是clone代碼，并且把submodules一起拉下來

流程3：構建項目

前提確認一下JDK 11+ & Maven 3.6+ 的環境，最好提前安裝node要求的版本

項目根目錄下執行./mvnw clean package -Dmaven.test.skip，構建項目較多，可能需要幾分鐘時間

這個步驟遇到比較多的問題，基本都是卡在node版本的下載和npm依賴包的拉取上，最終skywalking-ui的項目始終build不成功

node版本的下載，本地有8和18的版本，當前構建需要16的版本，下載網速出了問題

當前用到的v16.14.0，還可以確認一下node和npm的版本映射關系(https://nodejs.org/zh-cn/download/releases)，我當前在npm8.3.1，是可以兼容

npm install拉取依賴包也存在網速問題，可以替換鏡像源

skywalking/apm-webapp/pom.xml 中，找到

    <arguments>install --registry=https://registry.npmjs.org/</arguments><!--修改為淘寶鏡像--><arguments>install --registry=https://registry.npm.taobao.org/</arguments>

【qiang】的問題，以為會變快，其實反作用
整體流程構建如果最終都是卡在skywalking-ui前端項目的構建上，可以單獨使用對應的node版本去這個項目下單獨npm install & npm run build 確認下，避免要重頭再來，時間比較長

最終：

[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary for apm 9.6.0-SNAPSHOT:
[INFO] 
[INFO] apm ................................................ SUCCESS [  5.195 s]
[INFO] oap-protocol ....................................... SUCCESS [  0.335 s]
[INFO] apm-network ........................................ SUCCESS [ 28.284 s]
[INFO] oap-server ......................................... SUCCESS [  0.446 s]
[INFO] server-library ..................................... SUCCESS [  0.280 s]
[INFO] library-util ....................................... SUCCESS [  4.413 s]
[INFO] library-module ..................................... SUCCESS [  1.693 s]
[INFO] server-telemetry ................................... SUCCESS [  0.187 s]
[INFO] telemetry-api ...................................... SUCCESS [  1.580 s]
[INFO] server-testing ..................................... SUCCESS [  1.385 s]
[INFO] server-configuration ............................... SUCCESS [  0.372 s]
[INFO] configuration-api .................................. SUCCESS [  1.714 s]
[INFO] library-elasticsearch-client ....................... SUCCESS [  7.733 s]
[INFO] library-client ..................................... SUCCESS [  2.882 s]
[INFO] library-server ..................................... SUCCESS [  1.853 s]
[INFO] library-datacarrier-queue .......................... SUCCESS [  0.902 s]
[INFO] ai-pipeline ........................................ SUCCESS [  2.719 s]
[INFO] server-core ........................................ SUCCESS [ 27.428 s]
[INFO] library-kubernetes-support ......................... SUCCESS [  1.696 s]
[INFO] analyzer ........................................... SUCCESS [  0.509 s]
[INFO] meter-analyzer ..................................... SUCCESS [  4.938 s]
[INFO] agent-analyzer ..................................... SUCCESS [  4.672 s]
[INFO] log-analyzer ....................................... SUCCESS [  3.668 s]
[INFO] event-analyzer ..................................... SUCCESS [  1.611 s]
[INFO] server-receiver-plugin ............................. SUCCESS [  0.982 s]
[INFO] skywalking-sharing-server-plugin ................... SUCCESS [  1.546 s]
[INFO] skywalking-trace-receiver-plugin ................... SUCCESS [  2.440 s]
[INFO] zipkin-receiver-plugin ............................. SUCCESS [  1.786 s]
[INFO] skywalking-mesh-receiver-plugin .................... SUCCESS [  1.946 s]
[INFO] skywalking-management-receiver-plugin .............. SUCCESS [  1.607 s]
[INFO] skywalking-jvm-receiver-plugin ..................... SUCCESS [  1.963 s]
[INFO] receiver-proto ..................................... SUCCESS [ 17.060 s]
[INFO] envoy-metrics-receiver-plugin ...................... SUCCESS [  3.926 s]
[INFO] skywalking-clr-receiver-plugin ..................... SUCCESS [  1.542 s]
[INFO] skywalking-profile-receiver-plugin ................. SUCCESS [  1.549 s]
[INFO] otel-receiver-plugin ............................... SUCCESS [  2.132 s]
[INFO] skywalking-meter-receiver-plugin ................... SUCCESS [  1.976 s]
[INFO] skywalking-browser-receiver-plugin ................. SUCCESS [  2.540 s]
[INFO] skywalking-log-receiver-plugin ..................... SUCCESS [  2.236 s]
[INFO] configuration-discovery-receiver-plugin ............ SUCCESS [  2.265 s]
[INFO] skywalking-event-receiver-plugin ................... SUCCESS [  2.095 s]
[INFO] skywalking-zabbix-receiver-plugin .................. SUCCESS [  3.999 s]
[INFO] skywalking-ebpf-receiver-plugin .................... SUCCESS [  2.799 s]
[INFO] skywalking-telegraf-receiver-plugin ................ SUCCESS [  2.467 s]
[INFO] aws-firehose-receiver .............................. SUCCESS [  7.095 s]
[INFO] server-cluster-plugin .............................. SUCCESS [  0.148 s]
[INFO] cluster-zookeeper-plugin ........................... SUCCESS [  1.676 s]
[INFO] cluster-standalone-plugin .......................... SUCCESS [  1.344 s]
[INFO] cluster-kubernetes-plugin .......................... SUCCESS [  1.972 s]
[INFO] cluster-consul-plugin .............................. SUCCESS [  1.811 s]
[INFO] cluster-etcd-plugin ................................ SUCCESS [  1.955 s]
[INFO] cluster-nacos-plugin ............................... SUCCESS [  1.982 s]
[INFO] server-storage-plugin .............................. SUCCESS [  0.120 s]
[INFO] storage-jdbc-hikaricp-plugin ....................... SUCCESS [  5.366 s]
[INFO] storage-elasticsearch-plugin ....................... SUCCESS [  4.129 s]
[INFO] storage-banyandb-plugin ............................ SUCCESS [  4.208 s]
[INFO] oal-grammar ........................................ SUCCESS [  1.128 s]
[INFO] oal-rt ............................................. SUCCESS [  2.740 s]
[INFO] server-query-plugin ................................ SUCCESS [  0.134 s]
[INFO] zipkin-query-plugin ................................ SUCCESS [  2.032 s]
[INFO] server-fetcher-plugin .............................. SUCCESS [  0.111 s]
[INFO] kafka-fetcher-plugin ............................... SUCCESS [  3.037 s]
[INFO] server-health-checker .............................. SUCCESS [  1.661 s]
[INFO] mqe-grammar ........................................ SUCCESS [  0.735 s]
[INFO] mqe-rt ............................................. SUCCESS [  2.202 s]
[INFO] query-graphql-plugin ............................... SUCCESS [  3.807 s]
[INFO] promql-plugin ...................................... SUCCESS [  3.402 s]
[INFO] logql-plugin ....................................... SUCCESS [  1.966 s]
[INFO] server-alarm-plugin ................................ SUCCESS [  5.635 s]
[INFO] telemetry-prometheus ............................... SUCCESS [  1.485 s]
[INFO] exporter ........................................... SUCCESS [  3.927 s]
[INFO] grpc-configuration-sync ............................ SUCCESS [  2.922 s]
[INFO] configuration-apollo ............................... SUCCESS [  1.624 s]
[INFO] configuration-zookeeper ............................ SUCCESS [  1.661 s]
[INFO] configuration-etcd ................................. SUCCESS [  1.129 s]
[INFO] configuration-consul ............................... SUCCESS [  1.498 s]
[INFO] configuration-k8s-configmap ........................ SUCCESS [  1.837 s]
[INFO] configuration-nacos ................................ SUCCESS [  1.452 s]
[INFO] server-starter ..................................... SUCCESS [  7.200 s]
[INFO] server-tools ....................................... SUCCESS [  0.096 s]
[INFO] profile-exporter ................................... SUCCESS [  0.091 s]
[INFO] tool-profile-snapshot-server-mock .................. SUCCESS [  1.363 s]
[INFO] tool-profile-snapshot-bootstrap .................... SUCCESS [  2.893 s]
[INFO] tool-profile-snapshot-exporter ..................... SUCCESS [  3.118 s]
[INFO] data-generator ..................................... SUCCESS [  4.334 s]
[INFO] microbench ......................................... SUCCESS [  8.785 s]
[INFO] oap-server-bom ..................................... SUCCESS [  0.059 s]
[INFO] apm-webapp ......................................... SUCCESS [03:32 min]
[INFO] apache-skywalking-apm .............................. SUCCESS [ 37.877 s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  08:42 min
[INFO] Finished at: 2023-08-09T18:59:37+08:00
[INFO] ------------------------------------------------------------------------

流程4：構建產生的源代碼數據加入項目索引

IDEA中，可以找到對應的目錄，右鍵 “Mark Dictory as” -> “Generated Sources Root”，將構建產生的數據加入編輯器的索引中

apm-protocol/apm-network/target/generated-sources/protobuf
oap-server/server-core/target/generated-sources/protobuf
oap-server/server-receiver-plugin/receiver-proto/target/generated-sources/fbs
oap-server/server-receiver-plugin/receiver-proto/target/generated-sources/protobuf
oap-server/exporter/target/generated-sources/protobuf
oap-server/server-configuration/grpc-configuration-sync/target/generated-sources/protobuf
oap-server/server-alarm-plugin/target/generated-sources/protobuf
oap-server/oal-grammar/target/generated-sources

流程5：啟動前后端代碼

1.啟動OAP核心服務，接收數據上報與數據查詢

/skywalking/oap-server/server-starter/src/main/java/org/apache/skywalking/oap/server/starter/OAPServerStartUp.java
后端項目啟動，存儲的數據默認是H2，也可以自己根據標準版本部署對應的elastcisearch版本進行配置
由于整體是插件化的架構風格，在組件的選擇上使用 selector 即可快速鎖定選擇的組件。默認存儲是h2，修改為es的只要 selector: ${SW_STORAGE:elasticsearch} 并修改下方elasticsearch對應的配置即可

storage:selector: ${SW_STORAGE:h2} #--> 修改elasticsearch:namespace: ${SW_NAMESPACE:""}clusterNodes: ${SW_STORAGE_ES_CLUSTER_NODES:localhost:9200}protocol: ${SW_STORAGE_ES_HTTP_PROTOCOL:"http"}connectTimeout: ${SW_STORAGE_ES_CONNECT_TIMEOUT:3000}socketTimeout: ${SW_STORAGE_ES_SOCKET_TIMEOUT:30000}responseTimeout: ${SW_STORAGE_ES_RESPONSE_TIMEOUT:15000}numHttpClientThread: ${SW_STORAGE_ES_NUM_HTTP_CLIENT_THREAD:0}user: ${SW_ES_USER:""}password: ${SW_ES_PASSWORD:""}trustStorePath: ${SW_STORAGE_ES_SSL_JKS_PATH:""}trustStorePass: ${SW_STORAGE_ES_SSL_JKS_PASS:""}secretsManagementFile: ${SW_ES_SECRETS_MANAGEMENT_FILE:""} # Secrets management file in the properties format includes the username, password, which are managed by 3rd party tool.dayStep: ${SW_STORAGE_DAY_STEP:1} # Represent the number of days in the one minute/hour/day index.indexShardsNumber: ${SW_STORAGE_ES_INDEX_SHARDS_NUMBER:1} # Shard number of new indexesindexReplicasNumber: ${SW_STORAGE_ES_INDEX_REPLICAS_NUMBER:1} # Replicas number of new indexes# Specify the settings for each index individually.# If configured, this setting has the highest priority and overrides the generic settings.specificIndexSettings: ${SW_STORAGE_ES_SPECIFIC_INDEX_SETTINGS:""}# Super data set has been defined in the codes, such as trace segments.The following 3 config would be improve es performance when storage super size data in es.superDatasetDayStep: ${SW_STORAGE_ES_SUPER_DATASET_DAY_STEP:-1} # Represent the number of days in the super size dataset record index, the default value is the same as dayStep when the value is less than 0superDatasetIndexShardsFactor: ${SW_STORAGE_ES_SUPER_DATASET_INDEX_SHARDS_FACTOR:5} #  This factor provides more shards for the super data set, shards number = indexShardsNumber * superDatasetIndexShardsFactor. Also, this factor effects Zipkin traces.superDatasetIndexReplicasNumber: ${SW_STORAGE_ES_SUPER_DATASET_INDEX_REPLICAS_NUMBER:0} # Represent the replicas number in the super size dataset record index, the default value is 0.indexTemplateOrder: ${SW_STORAGE_ES_INDEX_TEMPLATE_ORDER:0} # the order of index templatebulkActions: ${SW_STORAGE_ES_BULK_ACTIONS:5000} # Execute the async bulk record data every ${SW_STORAGE_ES_BULK_ACTIONS} requestsbatchOfBytes: ${SW_STORAGE_ES_BATCH_OF_BYTES:10485760} # A threshold to control the max body size of ElasticSearch Bulk flush.# flush the bulk every 5 seconds whatever the number of requestsflushInterval: ${SW_STORAGE_ES_FLUSH_INTERVAL:5}concurrentRequests: ${SW_STORAGE_ES_CONCURRENT_REQUESTS:2} # the number of concurrent requestsresultWindowMaxSize: ${SW_STORAGE_ES_QUERY_MAX_WINDOW_SIZE:10000}metadataQueryMaxSize: ${SW_STORAGE_ES_QUERY_MAX_SIZE:10000}scrollingBatchSize: ${SW_STORAGE_ES_SCROLLING_BATCH_SIZE:5000}segmentQueryMaxSize: ${SW_STORAGE_ES_QUERY_SEGMENT_SIZE:200}profileTaskQueryMaxSize: ${SW_STORAGE_ES_QUERY_PROFILE_TASK_SIZE:200}profileDataQueryBatchSize: ${SW_STORAGE_ES_QUERY_PROFILE_DATA_BATCH_SIZE:100}oapAnalyzer: ${SW_STORAGE_ES_OAP_ANALYZER:"{\"analyzer\":{\"oap_analyzer\":{\"type\":\"stop\"}}}"} # the oap analyzer.oapLogAnalyzer: ${SW_STORAGE_ES_OAP_LOG_ANALYZER:"{\"analyzer\":{\"oap_log_analyzer\":{\"type\":\"standard\"}}}"} # the oap log analyzer. It could be customized by the ES analyzer configuration to support more language log formats, such as Chinese log, Japanese log and etc.advanced: ${SW_STORAGE_ES_ADVANCED:""}# Enable shard metrics and records indices into multi-physical indices, one index template per metric/meter aggregation function or record.logicSharding: ${SW_STORAGE_ES_LOGIC_SHARDING:false}# Custom routing can reduce the impact of searches. Instead of having to fan out a search request to all the shards in an index, the request can be sent to just the shard that matches the specific routing value (or values).enableCustomRouting: ${SW_STORAGE_ES_ENABLE_CUSTOM_ROUTING:false}h2:properties:jdbcUrl: ${SW_STORAGE_H2_URL:jdbc:h2:mem:skywalking-oap-db;DB_CLOSE_DELAY=-1;DATABASE_TO_UPPER=FALSE}dataSource.user: ${SW_STORAGE_H2_USER:sa}metadataQueryMaxSize: ${SW_STORAGE_H2_QUERY_MAX_SIZE:5000}maxSizeOfBatchSql: ${SW_STORAGE_MAX_SIZE_OF_BATCH_SQL:100}asyncBatchPersistentPoolSize: ${SW_STORAGE_ASYNC_BATCH_PERSISTENT_POOL_SIZE:1}

啟動后端服務跑在 127.0.0.1:11800

2.啟動web頁面項目，供頁面操作

/skywalking/apm-webapp/src/main/java/org/apache/skywalking/oap/server/webapp/ApplicationStartUp.java
啟動服務跑在 127.0.0.1:8080
瀏覽器中訪問，可以展示菜單頁

3.選擇一個springBoot項目

可以選擇下載skywalking-agent.jar，或者也走源碼構建的流程
下載地址：https://skywalking.apache.org/downloads/#JavaAgent
頁面選擇Java Agent，可以選擇源碼包構建，或者打包好的distribution的包
這邊直接下載打包好的包：https://www.apache.org/dyn/closer.cgi/skywalking/java-agent/8.16.0/apache-skywalking-java-agent-8.16.0.tgz
在springboot啟動參數中添加skywalking追蹤：-javaagent:~/Downloads/skywalking-agent/skywalking-agent.jar -Dskywalking.agent.service_name=demo -Dskywalking.collector.backend_service=127.0.0.1:11800
應用啟動后能夠查看到DB\REDIS相關的鏈接請求，訪問后端服務接口，幾秒后頁面能夠刷新報表。關注service,instance,endpoint相關指標

以上源碼構建，本地部署Skywalking項目就算告一段落啦