概述

為了提高系統運維和故障排查的效率， 日志系統采用 ELK（Elasticsearch、Logstash、Kibana）技術棧，通過 FileBeats 作為日志收集器，將來自不同節點的日志數據匯總并存儲在 Elasticsearch 中，最終通過 Kibana 進行可視化展示和分析。

系統配置包括 FileBeats 的收集規則、Logstash 的處理管道和 Elasticsearch 的索引設置。部署過程采用 Kubernetes 的 Helm Chart 實現，簡化了安裝和管理。

架構

FileBeats + Kafka + ELK

??： 在不影響舊有日志系統運行的前提下，我們在現有的 ELK+Kafka 架構基礎上，新增了 Filebeat 作為日志收集器，用于采集 Kubernetes 日志。

? 日志采集器Logstash其功能雖然強大，但是它依賴java、在數據量大的時候，Logstash進程會消耗過多的系統資源，這將嚴重影響業務系統的性能，而filebeat就是一個完美的替代者，它基于Go語言沒有任何依賴，配置文件簡單，格式明了，同時filebeat比logstash更加輕量級，所以占用系統資源極少，非常適合安裝在生產機器上。這就是推薦使用filebeat，也是 ELK Stack 在 Agent 的第一選擇。
? 此架構適合大型集群、海量數據的業務場景，它通過將前端Logstash Agent替換成filebeat，有效降低了收集日志對業務系統資源的消耗。同時，消息隊列使用kafka集群架構，有效保障了收集數據的安全性和穩定性，而后端Logstash和Elasticsearch均采用集群模式搭建，從整體上提高了ELK系統的高效性、擴展性和吞吐量。我所在的項目組采用的就是這套架構，由于生產所需的配置較高，且涉及較多持久化操作，采用的都是性能高配的云主機搭建方式而非時下流行的容器搭建

部署

elasticsearch

安裝方式：helm

安裝版本：8.5.1

?? helm安裝8版本elastic默認開啟ssl認證，我們這邊禁用認證

開始前奏操作，生成secret

# 根據提供的 elastic-certificates.p12 將 pcks12 中的信息分離出來，寫入文件
openssl pkcs12 -nodes -passin pass:'' -in elastic-certificates.p12 -out elastic-certificate.pem添加證書和密碼到集群
# 添加證書
kubectl create secret generic elastic-certificates --from-file=elastic-certificates.p12
kubectl create secret generic elastic-certificate-pem --from-file=elastic-certificate.pem# 設置集群用戶名密碼，用戶名不建議修改
##命令行
kubectl create secret generic elastic-credentials \--from-literal=username=elastic --from-literal=password=123456
##yaml
apiVersion: v1
data:password: MTIzNDU2username: ZWxhc3RpYw==
kind: Secret
metadata:name: elastic-credentialsnamespace: elastic
type: Opaque

🏴 加油特種兵

先添加elasticsearch的helm倉庫，并拉取對應版本的Charts并對其解壓

[root@ycloud ~]# helm repo add elastic https://helm.elastic.co
"elastic" has been added to your repositories
[root@ycloud ~]# helm pull elastic/elasticsearch --version 8.5.1
[root@ycloud ~]# tar -zxvf elasticsearch-8.5.1.tgz 
elasticsearch/Chart.yaml
elasticsearch/values.yaml
......

調整values

replicas: 2
minimumMasterNodes: 1esMajorVersion: ""# Allows you to add any config files in /usr/share/elasticsearch/config/
# such as elasticsearch.yml and log4j2.properties# ==================安全配置========================
protocol: http###掛在證書，這里用我們上面創建的證書
secretMounts: - name: elastic-certificatessecretName: elastic-certificatespath: /usr/share/elasticsearch/config/certsdefaultMode: 0755esConfig: elasticsearch.yml: |xpack.security.enabled: truexpack.security.transport.ssl.enabled: truexpack.security.transport.ssl.verification_mode: certificatexpack.security.transport.ssl.keystore.path: /usr/share/elasticsearch/config/certs/elastic-certificates.p12xpack.security.transport.ssl.truststore.path: /usr/share/elasticsearch/config/certs/elastic-certificates.p12
#    key:
#      nestedkey: value
#  log4j2.properties: |
#    key = value## 關閉ssl認證
createCert: false        
esJavaOpts: "-Xmx1g -Xms1g"###指定elastic用戶密碼
extraEnvs: - name: ELASTIC_USERNAMEvalueFrom:secretKeyRef:name: elastic-credentials    ###用我們創建的secretkey: username- name: ELASTIC_PASSWORDvalueFrom:secretKeyRef:name: elastic-credentials		###用我們創建的secretkey: password
###true開啟持久化
persistence:enabled: true

kibana

安裝方式：helm

安裝版本：8.5.1

??: 默認使用身份驗證和TLS部署Kibana 8.5.1連接到Elasticsearch

🐤:重置kibana_system賬戶密碼

[root@ycloud hub]# kubectl exec -it elasticsearch-master-0 -- bin/elasticsearch-reset-password -u kibana_system
Defaulted container "elasticsearch" out of: elasticsearch, configure-sysctl (init)
This tool will reset the password of the [kibana_system] user to an autogenerated value.
The password will be printed in the console.
Please confirm that you would like to continue [y/N]yPassword for the [kibana_system] user successfully reset.
New value: nVUE*km=842x7duqB4TR

🐤:更新 elastic-credentials文件

# [root@ycloud hub]# echo nVUE*km=842x7duqB4TR|base64 
blZVRSprbT04NDJ4N2R1cUI0VFIK
apiVersion: v1
data:kibana_password: blZVRSprbT04NDJ4N2R1cUI0VFIK kibana_username: a2liYW5hX3N5c3RlbQ==password: MTIzNDU2username: ZWxhc3RpYw==
kind: Secret
metadata:name: elastic-credentialsnamespace: elastic
type: Opaque

🏴 加油特種兵

先添加kibana的helm倉庫，并拉取對應版本的Charts并對其解壓

[root@ycloud ~]# helm repo add elastic https://helm.elastic.co
"elastic" has been added to your repositories
[root@ycloud ~]# helm pull elastic/kibana --version 8.5.1
[root@ycloud ~]# tar -zxvf kibana-8.5.1.tgz 
kibana/Chart.yaml
kibana/values.yaml
......

調整template

# cd kibana/templates/ ； rm -rf post-delete-* pre-install-*
##刪除deployment.yaml部分內容
---- name: ELASTICSEARCH_SSL_CERTIFICATEAUTHORITIESvalue: "{{ template "kibana.home_dir" . }}/config/certs/{{ .Values.elasticsearchCertificateAuthoritiesFile }}"- name: ELASTICSEARCH_SERVICEACCOUNTTOKENvalueFrom:secretKeyRef:name: {{ template "kibana.fullname" . }}-es-tokenkey: tokenoptional: false
--- readinessProbe:
{{ toYaml .Values.readinessProbe | indent 10 }}exec:command:- bash- -c- |#!/usr/bin/env bash -e# Disable nss cache to avoid filling dentry cache when calling curl# This is required with Kibana Docker using nss < 3.52export NSS_SDB_USE_CACHE=nohttp () {local path="${1}"set -- -XGET -s --fail -Lif [ -n "${ELASTICSEARCH_USERNAME}" ] && [ -n "${ELASTICSEARCH_PASSWORD}" ]; thenset -- "$@" -u "${ELASTICSEARCH_USERNAME}:${ELASTICSEARCH_PASSWORD}"fiSTATUS=$(curl --output /dev/null --write-out "%{http_code}" -k "$@" "{{ .Values.protocol }}://localhost:{{ .Values.httpPort }}${path}")if [[ "${STATUS}" -eq 200 ]]; thenexit 0fiecho "Error: Got HTTP code ${STATUS} but expected a 200"exit 1}http "{{ .Values.healthCheckPath }}"
---- name: elasticsearch-certssecret:secretName: {{ .Values.elasticsearchCertificateSecret }}
---- name: elasticsearch-certsmountPath: {{ template "kibana.home_dir" . }}/config/certsreadOnly: true
---

調整values

# cd kibana/templates/
# rm -rf post-delete-* pre-install-*
---
elasticsearchHosts: "http://elasticsearch-master:9200"
#elasticsearchCertificateSecret: elasticsearch-master-certs
#elasticsearchCertificateAuthoritiesFile: ca.crt
elasticsearchCredentialSecret: elastic-credentialsreplicas: 1extraEnvs:- name: "NODE_OPTIONS"value: "--max-old-space-size=1800"- name: 'ELASTICSEARCH_USERNAME'valueFrom:secretKeyRef:name: elastic-credentialskey: kibana_username- name: 'ELASTICSEARCH_PASSWORD'valueFrom:secretKeyRef:name: elastic-credentialskey: kibana_passwordresources:requests:cpu: "1000m"memory: "2Gi"limits:cpu: "1000m"memory: "2Gi"protocol: httpserverHost: "0.0.0.0"healthCheckPath: "/app/kibana"

filebeat

??: 詳情見 Here

總結

日志系統已成功應用于多種業務場景。為了進一步提高系統的穩定性，后面我們在新的部署中采用 ClickHouse 作為日志存儲解決方案。