官網提供yaml地址下載部署
https://doris.apache.org/zh-CN/docs/2.0/install/cluster-deployment/k8s-deploy/install-env/
禁用和關閉 swap
在部署 Doris 時，建議關閉 swap 分區。
通過以下命令可以永久關閉 swap 分區。
echo "vm.swappiness = 0">> /etc/sysctl.conf
swapoff -a && swapon -a
sysctl -p
設置系統最大打開文件句柄數
vi /etc/security/limits.conf
* soft nofile 65536
* hard nofile 65536
修改虛擬內存區域數量
修改虛擬內存區域至少 2000000
sysctl -w vm.max_map_count=2000000
關閉透明大頁
在部署 Doris 時，建議關閉透明大頁。
echo never > /sys/kernel/mm/transparent_hugepage/enabled
echo never > /sys/kernel/mm/transparent_hugepage/defrag
部署 Doris Operator
添加 Doris Cluster 資源定義
Doris Operator 使用自定義資源定義（Custom Resource Definition, CRD）擴展 Kubernetes。Doris Cluster 的 CRD 中封裝了對 Doris 對象的描述，例如對 FE 或 BE 的描述，詳細內容可以參考 doris-operator-api。在部署 Doris 前，需要先創建 Doris Cluster 的 CRD。
通過以下命令可以在 Kubernetes 環境中部署 Doris Cluster CRD：
kubectl create -f https://raw.githubusercontent.com/apache/doris-operator/master/config/crd/bases/doris.selectdb.com_dorisclusters.yaml
如果沒有外網，先將 CRD 文件下載到本地：
wget https://raw.githubusercontent.com/apache/doris-operator/master/config/crd/bases/doris.selectdb.com_dorisclusters.yaml
kubectl create -f ./doris.selectdb.com_dorisclusters.yaml
以下是期望輸出結果：
customresourcedefinition.apiextensions.k8s.io/dorisclusters.doris.selectdb.com created
在創建了 Doris Cluster CRD 后，可以通過以下命令查看創建的 CRD。
kubectl get crd | grep doris
以下為期望輸出結果：
dorisclusters.doris.selectdb.com 2024-02-22T16:23:13Z
添加 Doris Operator
方案一：快速部署 Doris Operator
可以直接拉去倉庫中的 Doris Operator 模板進行快速部署。
使用以下命令可以在 Kubernetes 集群中部署 Doris Operator：
kubectl apply -f https://raw.githubusercontent.com/apache/doris-operator/master/config/operator/operator.yaml
以下為期望輸出結果：
namespace/doris created
role.rbac.authorization.k8s.io/leader-election-role created
rolebinding.rbac.authorization.k8s.io/leader-election-rolebinding created
clusterrole.rbac.authorization.k8s.io/doris-operator created
clusterrolebinding.rbac.authorization.k8s.io/doris-operator-rolebinding created
serviceaccount/doris-operator created
deployment.apps/doris-operator created
方案二：自定義部署 Doris Operator
在創建完 CRD 后，在 Kubernetes 集群上部署 Doris Operator 有兩種方式：在線與離線部署。
在 operator.yaml 文件中規范了部署 operator 的服務的最低要求。為了適配復雜的生產環境，可以下載 operator.yaml 文件后，按照期望更新其中配置。
在線安裝 Doris Operator
在修改 operator.yaml 文件后，可以使用以下命令部署 Doris Operator 服務：
kubectl apply -f /data/k8s-yaml/doris/operator.yaml
以下為期望輸出結果：
namespace/doris created
role.rbac.authorization.k8s.io/leader-election-role created
rolebinding.rbac.authorization.k8s.io/leader-election-rolebinding created
clusterrole.rbac.authorization.k8s.io/doris-operator created
clusterrolebinding.rbac.authorization.k8s.io/doris-operator-rolebinding created
serviceaccount/doris-operator created
deployment.apps/doris-operator created
離線安裝 Doris Operator
下載 operator 運行所需鏡像文件
如果服務器沒有連通外網，需要先下載對應的 operator 鏡像文件。Doris Operator 用到以下的鏡像：
selectdb/doris.k8s-operator:latest
在可以連通外網的服務器中運行以下的命令，可以將鏡像下載下來：
download doris operator image
docker pull selectdb/doris.k8s-operator:latest
save the doris operator image as a tar package
docker save -o doris.k8s-operator-latest.tar selectdb/doris.k8s-operator:latest
將已打包的 tar 文件放置到所有的 Kubernetes node 節點中，運行以下命令上傳鏡像：
docker load -i doris.k8s-operator-latest.tar
配置 Doris Operator
下載 operator.yaml 文件后，可以根據生產環境期望修改模板。
Doris Operator 在 Kubernetes 集群中是一個無狀態的 Deployment，可以根據需求修改如 limits、replica、label、namespace 等項目。如需要指定某一版本的 doirs operator 鏡像，可以在上傳鏡像后對 operator.yaml 文件做如下修改：
...
containers:- command:- /dorisoperatorargs:- --leader-electimage: selectdb/doris.k8s-operator:v1.0.0name: dorisoperatorsecurityContext:allowPrivilegeEscalation: falsecapabilities:drop:- "ALL"...
安裝 Doris Operator
在修改 Doris Operator 模板后，可以使用 apply 命令部署 Operator：
kubectl apply -f ./data/k8s-yaml/doris/operator.yaml
第三種方式部署
根據服務器環境，修改doris_be.yml文件。
apiVersion: v1
kind: Service
metadata:name: doris-be-cluster1labels:app: doris-be-cluster1
spec:ports:- port: 9060name: be-port- port: 8040name: webserver-port- port: 9050name: heartbeat-port #This name should be fixed. Doris will get the port information through this name- port: 8060name: brpc-portclusterIP: Noneselector:app: doris-be-cluster1
---
apiVersion: v1
kind: Service
metadata:name: doris-be-cluster1labels:app: doris-be-cluster1
spec:ports:- port: 9060name: be-port- port: 8040name: webserver-port- port: 9050name: heartbeat-port #This name should be fixed. Doris will get the port information through this name- port: 8060name: brpc-portclusterIP: Noneselector:app: doris-be-cluster1
---
apiVersion: apps/v1
kind: StatefulSet
metadata:name: doris-be-cluster1labels:app: doris-be-cluster1
spec:selector:matchLabels:app: doris-be-cluster1serviceName: doris-be-cluster1replicas: 3template:metadata:name: doris-be-cluster1labels:app: doris-be-cluster1spec:containers:- name: doris-be-cluster1#Need to change to real mirror information#image: apache-doris-be:test# 修改點1： 鏡像地址修改為真實doris鏡像，可在https://hub.docker.com/r/apache/doris/tags找到需要的鏡像版本image: apache/doris:2.0.0_alpha-be-x86_64imagePullPolicy: IfNotPresent#節點選擇nodeSelector:node: middlewareenv:#Specify the startup type as k8s to bypass some restrictions of the official image initialization script- name: BUILD_TYPEvalue: "k8s"# 修改點2： 增加環境變量，寫明FE的IP與端口- name: FE_MASTER_IPvalue: "doris-follower-cluster1-0.doris-follower-cluster1.doris.svc.cluster.local"- name: FE_MASTER_PORTvalue: "9030"ports:- containerPort: 9060name: be-port- containerPort: 8040name: webserver-port- containerPort: 9050name: heartbeat-port- containerPort: 8060name: brpc-portvolumeMounts:#Mount the configuration file in the way of configmap- name: confmountPath: /data/apache-doris/be/conf#Ifnot mounted, when enable_profile, error will be reported when querying the data from jdbc catalog#Error message: error setting certificate verify locations: CAfile:/etc/pki/tls/certs/ca-bundle.crt CApath: none- name: sysmountPath: /etc/pki# 修改點3： 掛載存儲subPath: pkireadOnly: true# 同修改點3- name: sysmountPath: /data/apache-doris/be/storagesubPath: storagevolumes:- name: confconfigMap:name: be-conf- name: sys# 修改點4：不使用hostpath，注釋掉#hostPath:#path: /etc/pki# 修改點5： 增加存儲配置，此處使用nfs-dynamic-class或者longhornvolumeClaimTemplates:- metadata:name: sysspec:storageClassName: nfs-dynamic-classaccessModes: [ "ReadWriteOnce" ]resources:requests:storage: 50Gi
---
apiVersion: v1
kind: ConfigMap
metadata:name: be-conf
data:be.conf: |PPROF_TMPDIR="$DORIS_HOME/log/"sys_log_level = INFObe_port = 9060webserver_port = 8040heartbeat_service_port = 9050brpc_port = 8060# 修改點6： 修改網段為k8s使用網段，配置數據存儲路徑priority_networks = 10.44.0.0/16storage_root_path = /data/apache-doris/be/storage
根據服務器環境，修改doris_follower.yml文件
apiVersion: v1
kind: Service
metadata:name: doris-follower-cluster1labels:app: doris-follower-cluster1
spec:ports:- port: 8030name: http-port- port: 9020name: rpc-port- port: 9030name: query-port- port: 9010name: edit-log-port #This name should be fixed. Doris will get the port information through this nameclusterIP: Noneselector:app: doris-follower-cluster1
---
apiVersion: v1
kind: Service
metadata:name: doris-follower-cluster1labels:app: doris-follower-cluster1
spec:ports:- port: 8030name: http-port- port: 9020name: rpc-port- port: 9030name: query-port- port: 9010name: edit-log-port #This name should be fixed. Doris will get the port information through this nameclusterIP: Noneselector:app: doris-follower-cluster1
---
apiVersion: apps/v1
kind: StatefulSet
metadata:name: doris-follower-cluster1labels:app: doris-follower-cluster1
spec:selector:matchLabels:app: doris-follower-cluster1serviceName: doris-follower-cluster1# 修改點1： 修改fe副本為1replicas: 1template:metadata:name: doris-follower-cluster1labels:app: doris-follower-cluster1spec:containers:- name: doris-follower-cluster1#Need to change to real mirror information# 修改點2： 鏡像地址修改為真實doris鏡像，可在https://hub.docker.com/r/apache/doris/tags找到需要的鏡像版本image: apache/doris:2.0.0_alpha-fe-x86_64imagePullPolicy: IfNotPresent#節點選擇nodeSelector:node: middlewareenv:# 修改點3： 增加了APP_NAMESPACE與FE_IPADDRESS環境變量- name: APP_NAMESPACEvalueFrom:fieldRef:fieldPath: metadata.namespace- name: FE_IPADDRESSvalueFrom:fieldRef:fieldPath: status.podIP#Specify the startup type as k8s to bypass some restrictions of the official image initialization script- name: BUILD_TYPEvalue: "k8s"#Initialize the fe of three nodes- name: FE_INIT_NUMBER# 修改點4： 將數量改為1value: "1"#ServiceName of bakend_cn node,（if do not have bakend_cn node,do not configure this environment variable）# 修改點5： 不使用cn節點，注釋變量CN_SERVICE CN_STATEFULSET#- name: CN_SERVICE# value: "doris-cn-cluster1"#StatefulSetName of bakend_cn node,（if do not have bakend_cn node,do not configure this environment variable)#- name: CN_STATEFULSET# value: "doris-cn-cluster1"#ServiceName of bakend node,（if do not have bakend node,do not configure this environment variable）- name: BE_SERVICEvalue: "doris-be-cluster1"#StatefulSetName of bakend node,（if do not have bakend node,do not configure this environment variable)- name: BE_STATEFULSETvalue: "doris-be-cluster1"#ServiceName of follower node,（if do not have follower node,do not configure this environment variable）- name: FE_SERVICEvalue: "doris-follower-cluster1"##StatefulSetName of follower node,（if do not have follower node,do not configure this environment variable)- name: FE_STATEFULSETvalue: "doris-follower-cluster1"ports:- containerPort: 8030name: http-port- containerPort: 9020name: rpc-port- containerPort: 9030name: query-port- containerPort: 9010name: edit-log-portvolumeMounts:#Mount the configuration file in the way of configmap- name: confmountPath: /data/apache-doris/fe/conf#In order to call the api of k8s- name: kube# 使用本地配置則為/root/.kube/configmountPath: /root/.kubereadOnly: true# 修改點6： 配置存儲，用于元數據持久化- name: metadatamountPath: /data/apache-doris/fe/doris-metavolumes:- name: confconfigMap:name: follower-conf- name: kube# 修改點7： 修改為使用configMap(此處可以不修改，使用本地配置)#hostPath:#path: /root/.kube/configconfigMap:name: kube-conf# 修改點8： 增加存儲配置，此處使用nfs-dynamic-class 或者 longhornvolumeClaimTemplates:- metadata:name: metadataspec:storageClassName: nfs-dynamic-class accessModes: [ "ReadWriteOnce" ]resources:requests:storage: 10Gi
---
apiVersion: v1
kind: ConfigMap
metadata:name: follower-conf
data:fe.conf: |# 修改點9： 修改網段為k8s使用網段priority_networks = 10.44.0.0/16#It can automatically maintain node information by getting the number of replicas of StatefulSet, similar to alter system add/drop backenable_deploy_manager = k8s#Automatically adjust the IP of the node according to the domain name (for example, after the pod is restarted, the domain name is still doris-be-cluster1-0-doris-be-cluster1.default.svc.cluster.local, but the IP may change from 172.16.0.9 to 172.16.0.10)enable_fqdn_mode = trueLOG_DIR = ${DORIS_HOME}/logsys_log_level = INFOhttp_port = 8030rpc_port = 9020query_port = 9030edit_log_port = 9010#Doris needs to generate the log4j configuration file according to the fe.yml configuration information, which is written in the same directory as fe.yml by default, but the config we mount is readonly, so specify this configuration to write the log4j file to another locationcustom_config_dir = /data/apache-doris/#when set to false, the backend will not be dropped and remaining in DECOMMISSION statedrop_backend_after_decommission = false# 修改點10： 增加元數據、java等配置mysql_service_nio_enabled = trueJAVA_OPTS = "-Xmx8192m -XX:+UseMembar -XX:SurvivorRatio=8 -XX:MaxTenuringThreshold=7 -XX:+PrintGCDateStamps -XX:+PrintGCDetails -XX:+UseConcMarkSweepGC -XX:+UseParNewGC -XX:+CMSClassUnloadingEnabled -XX:-CMSParallelRemarkEnabled -XX:CMSInitiatingOccupancyFraction=80 -XX:SoftRefLRUPolicyMSPerMB=0 -Xloggc:$DORIS_HOME/log/fe.gc.log.$DATE"JAVA_OPTS_FOR_JDK_9 = "-Xmx8192m -XX:SurvivorRatio=8 -XX:MaxTenuringThreshold=7 -XX:+CMSClassUnloadingEnabled -XX:-CMSParalle=80 -XX:SoftRefLRUPolicyMSPerMB=0 -Xlog:gc*:$DORIS_HOME/log/fe.gc.log.$DATE:time"meta_dir = /data/apache-doris/fe/doris-meta#metadata_failure_recovery = true
外部訪問sevice
kind: Service
apiVersion: v1
metadata:name: doris-follower-loacllabels:app: doris-follower-loacl
spec:ports:- name: httpprotocol: TCPport: 8030targetPort: 8030nodePort: 28030- name: tcpprotocol: TCPport: 9030targetPort: 9030nodePort: 29030selector:app: doris-follower-cluster1type: NodePort
部署
kubectl create ns doris
kubectl apply -f doris_be.yml -n doris
kubectl apply -f doris_follower.yml -n doris
kubectl apply -f doris-svc.yaml -n doris
- 訪問及使用, 默認賬號為root，無密碼
# web端訪問地址
http://[節點IP]:20803/login
# 使用mysql client連接地址
host: [節點IP]
port: 29030
user: root
pass:
# 使用mysql client連接后可修改root密碼
SET PASSWORD FOR 'root' = PASSWORD('your_password');
