關鍵字 : CNI calico vxlan flannel ipv6-only ipv6+ipv4

在搭建ipv6-only或ipv6+ipv4的k8s集群時，在worker節點加入集群后，發現worker節點上的CNI啟動失敗。

以下是calico的啟動失敗情況 :

kubectl get pod -A

輸出如下 :

NAMESPACE     NAME                                      READY   STATUS                  RESTARTS      AGE
kube-system   calico-kube-controllers-79949b87d-ptq2r   1/1     Running                 0             19m
kube-system   calico-node-jbrn7                         0/1     Init:CrashLoopBackOff   7 (40s ago)   14m
kube-system   calico-node-xnwdx                         1/1     Running                 0             19m
kube-system   coredns-6766b7b6bb-wc5j5                  1/1     Running                 0             20m
kube-system   coredns-6766b7b6bb-wvg5w                  1/1     Running                 0             20m
kube-system   etcd-myserver1                            1/1     Running                 0             20m
kube-system   kube-apiserver-myserver1                  1/1     Running                 0             20m
kube-system   kube-controller-manager-myserver1         1/1     Running                 0             20m
kube-system   kube-proxy-g8gxb                          1/1     Running                 0             20m
kube-system   kube-proxy-lnddv                          1/1     Running                 0             14m
kube-system   kube-scheduler-myserver1                  1/1     Running                 0             20m

查看POD calico-node-jbrn7的詳細情況，輸出類似如下 :

kubectl describe pod -n kube-system calico-node-jbrn7

輸出如下 :

Events:Type     Reason     Age                  From               Message----     ------     ----                 ----               -------Normal   Scheduled  20m                  default-scheduler  Successfully assigned kube-system/calico-node-jbrn7 to worker1Normal   Pulled     20m                  kubelet            Container image "docker.io/calico/cni:v3.29.3" already present on machineNormal   Created    20m                  kubelet            Created container: upgrade-ipamNormal   Started    20m                  kubelet            Started container upgrade-ipamNormal   Created    6m40s (x8 over 20m)  kubelet            Created container: install-cniNormal   Started    6m40s (x8 over 20m)  kubelet            Started container install-cniNormal   Pulled     68s (x9 over 20m)    kubelet            Container image "docker.io/calico/cni:v3.29.3" already present on machineWarning  BackOff    11s (x76 over 20m)   kubelet            Back-off restarting failed container install-cni in pod calico-node-jbrn7_kube-system(a888f1ad-ec45-4207-94ac-f2953bda9d0e)

在事件Events中可以看到是執行POD中的名為install-cni的容器時發生了異常.

再查看容器install-cni的日志可以看到如下內容 :

2025-05-29 09:53:04.523 [INFO][1] cni-installer/install.go 233: CNI plugin version: v3.29.3
2025-05-29 09:53:04.523 [INFO][1] cni-installer/install.go 185: /host/secondary-bin-dir is not writeable, skipping
2025-05-29 09:53:04.523 [WARNING][1] cni-installer/winutils.go 150: Neither --kubeconfig nor --master was specified.  Using the inClusterConfig.  This might not work.
2025-05-29 09:53:34.524 [ERROR][1] cni-installer/token_watch.go 108: Unable to create token for CNI kubeconfig error=Post "https://[fd15:4ba5:5a2b:1008:2000::1]:443/api/v1/namespaces/kube-system/serviceaccounts/calico-cni-plugin/token": dial tcp [fd15:4ba5:5a2b:1008:2000::1]:443: i/o timeout
2025-05-29 09:53:34.524 [FATAL][1] cni-installer/install.go 478: Unable to create token for CNI kubeconfig error=Post "https://[fd15:4ba5:5a2b:1008:2000::1]:443/api/v1/namespaces/kube-system/serviceaccounts/calico-cni-plugin/token": dial tcp [fd15:4ba5:5a2b:1008:2000::1]:443: i/o timeout

即：無法連接到API Server的clusterIP [fd15:4ba5:5a2b:1008:2000::1]:443
這種情況在IPV4時不會出現

原理不多說，直接給出解決辦法:

• 修改calico的YAML文件,讓calico-node連接API SERVER的物理IPV6地址,即執行ip a所看到的IPV6地址

在calico.yaml文件中新增名為kubernetes-services-endpoint的ConfigMap對象，如下所示:

kind: ConfigMap
apiVersion: v1
metadata:name: kubernetes-services-endpointnamespace: kube-system
data:# 指定 API Server 的節點 IPKUBERNETES_SERVICE_HOST: "fd15:4ba5:5a2b:1008:192:168:186:40"KUBERNETES_SERVICE_PORT: "6443"

注意 :

• 1.必須新創建ConfigMap對象，不能在原有的名為calico-config的ConfigMap對象上修改;
• 2.新創建ConfigMap對象的名字必須是kubernetes-services-endpoint
• 3.在IPV6單棧和IPV6為主的雙棧情況下還需要在calico.yaml中的DaemonSet設置其它相關環境變量，這里就不贅述了

下面是flannel的修改方法
修改kube-flannel.yml中的DaemonSet下的env部分，新增環境變量KUBERNETES_SERVICE_HOST和KUBERNETES_SERVICE_PORT.如下所示 :

---
apiVersion: apps/v1
kind: DaemonSet
metadata:labels:app: flannelk8s-app: flanneltier: nodename: kube-flannel-dsnamespace: kube-flannel
spec:selector:matchLabels:app: flannelk8s-app: flanneltemplate:metadata:labels:app: flannelk8s-app: flanneltier: nodespec:affinity:nodeAffinity:requiredDuringSchedulingIgnoredDuringExecution:nodeSelectorTerms:- matchExpressions:- key: kubernetes.io/osoperator: Invalues:- linuxcontainers:- args:- --ip-masq- --kube-subnet-mgrcommand:- /opt/bin/flanneldenv:- name: POD_NAMEvalueFrom:fieldRef:fieldPath: metadata.name- name: POD_NAMESPACEvalueFrom:fieldRef:fieldPath: metadata.namespace- name: EVENT_QUEUE_DEPTHvalue: "5000"- name: FLANNELD_IFACEvalue: "ens33"# 指定API Server的節點IP地址- name: KUBERNETES_SERVICE_HOSTvalue: "fd15:4ba5:5a2b:1008:192:168:186:40"- name: KUBERNETES_SERVICE_PORTvalue: "6443"image: ghcr.io/flannel-io/flannel:v0.26.7name: kube-flannel