【0】資源配置文件
[root@mcwk8s03 mcwtest]# ls
mcwdeploy.yaml
[root@mcwk8s03 mcwtest]# cat mcwdeploy.yaml
apiVersion: apps/v1
kind: Deployment
metadata:labels:app: mcwpythonname: mcwtest-deploy
spec:replicas: 1selector:matchLabels:app: mcwpythontemplate:metadata:labels:app: mcwpythonspec:containers:- command:- sh- -c- echo 123 >>/mcw.txt && cd / && rm -rf /etc/yum.repos.d/* && curl -o /etc/yum.repos.d/CentOS-Base.repo https://mirrors.aliyun.com/repo/Centos-vault-8.5.2111.repo && yum install -y python2 && python2 -m SimpleHTTPServer 20000image: centosimagePullPolicy: IfNotPresentname: mcwtestdnsPolicy: "None"dnsConfig:nameservers:- 8.8.8.8- 8.8.4.4searches:#- namespace.svc.cluster.local- my.dns.search.suffixoptions:- name: ndotsvalue: "5"
---
apiVersion: v1
kind: Service
metadata:name: mcwtest-svc
spec:ports:- name: mcwportport: 2024protocol: TCPtargetPort: 20000selector:app: mcwpythontype: NodePort
[root@mcwk8s03 mcwtest]#
[1]查看服務部分通,部分不通
[root@mcwk8s03 mcwtest]# kubectl get svc
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
kubernetes ClusterIP 10.2.0.1 <none> 443/TCP 583d
mcwtest-svc NodePort 10.2.0.155 <none> 2024:41527/TCP 133m
nginx ClusterIP None <none> 80/TCP 413d
[root@mcwk8s03 mcwtest]# curl -I 10.2.0.155
curl: (7) Failed connect to 10.2.0.155:80; Connection timed out
[root@mcwk8s03 mcwtest]# curl -I 10.0.0.33:41527
curl: (7) Failed connect to 10.0.0.33:41527; Connection refused
[root@mcwk8s03 mcwtest]# curl -I 10.0.0.36:41527
curl: (7) Failed connect to 10.0.0.36:41527; Connection timed out
[root@mcwk8s03 mcwtest]# curl -I 10.0.0.35:41527
HTTP/1.0 200 OK
Server: SimpleHTTP/0.6 Python/2.7.18
Date: Tue, 04 Jun 2024 16:38:18 GMT
Content-type: text/html; charset=ANSI_X3.4-1968
Content-Length: 816[root@mcwk8s03 mcwtest]#
【2】查看,能通的IP,是因為在容器所在的宿主機,因此需要排查容器跨宿主機是否可以通信
[root@mcwk8s03 mcwtest]# kubectl get pod -o wide|grep mcwtest
mcwtest-deploy-6465665557-g9zjd 1/1 Running 0 37m 172.17.98.13 mcwk8s05 <none> <none>
[root@mcwk8s03 mcwtest]# kubectl get nodes -o wide
NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME
mcwk8s05 Ready <none> 580d v1.15.12 10.0.0.35 <none> CentOS Linux 7 (Core) 3.10.0-693.el7.x86_64 docker://20.10.21
mcwk8s06 Ready <none> 580d v1.15.12 10.0.0.36 <none> CentOS Linux 7 (Core) 3.10.0-693.el7.x86_64 docker://20.10.21
[root@mcwk8s03 mcwtest]#
【3】排查容器是否可以跨宿主機IP,首先這個容器是在這個宿主機上,應該優先排查這個宿主機是否能到其他機器的docker0 IP
[root@mcwk8s03 mcwtest]# ifconfig docker
docker0: flags=4099<UP,BROADCAST,MULTICAST> mtu 1500inet 172.17.83.1 netmask 255.255.255.0 broadcast 172.17.83.255ether 02:42:e9:a4:51:4f txqueuelen 0 (Ethernet)RX packets 0 bytes 0 (0.0 B)RX errors 0 dropped 0 overruns 0 frame 0TX packets 0 bytes 0 (0.0 B)TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0[root@mcwk8s03 mcwtest]# [root@mcwk8s05 /]# ifconfig docker
docker0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1450inet 172.17.98.1 netmask 255.255.255.0 broadcast 172.17.98.255inet6 fe80::42:18ff:fee1:e8fc prefixlen 64 scopeid 0x20<link>ether 02:42:18:e1:e8:fc txqueuelen 0 (Ethernet)RX packets 548174 bytes 215033771 (205.0 MiB)RX errors 0 dropped 0 overruns 0 frame 0TX packets 632239 bytes 885330301 (844.3 MiB)TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0[root@mcwk8s05 /]# [root@mcwk8s06 ~]# ifconfig docker
docker0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1450inet 172.17.9.1 netmask 255.255.255.0 broadcast 172.17.9.255inet6 fe80::42:f0ff:fefa:133e prefixlen 64 scopeid 0x20<link>ether 02:42:f0:fa:13:3e txqueuelen 0 (Ethernet)RX packets 229 bytes 31724 (30.9 KiB)RX errors 0 dropped 0 overruns 0 frame 0TX packets 212 bytes 53292 (52.0 KiB)TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0[root@mcwk8s06 ~]# 可以看到,05宿主機是不通其他機器的docker0的,03 06是互通的
[root@mcwk8s05 /]# ping -c 1 172.17.83.1
PING 172.17.83.1 (172.17.83.1) 56(84) bytes of data.
^C
--- 172.17.83.1 ping statistics ---
1 packets transmitted, 0 received, 100% packet loss, time 0ms[root@mcwk8s05 /]# ping -c 1 172.17.9.1
PING 172.17.9.1 (172.17.9.1) 56(84) bytes of data.
^C
--- 172.17.9.1 ping statistics ---
1 packets transmitted, 0 received, 100% packet loss, time 0ms[root@mcwk8s05 /]# [root@mcwk8s06 ~]# ping -c 1 172.17.83.1
PING 172.17.83.1 (172.17.83.1) 56(84) bytes of data.
64 bytes from 172.17.83.1: icmp_seq=1 ttl=64 time=0.246 ms--- 172.17.83.1 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.246/0.246/0.246/0.000 ms
[root@mcwk8s06 ~]#
【4】可以看到,etcd里面,沒有005宿主機的docker網段的
[root@mcwk8s03 mcwtest]# etcdctl ls /coreos.com/network/subnets
/coreos.com/network/subnets/172.17.9.0-24
/coreos.com/network/subnets/172.17.83.0-24
[root@mcwk8s03 mcwtest]# etcdctl get /coreos.com/network/subnets/172.17.9.0-24
{"PublicIP":"10.0.0.36","BackendType":"vxlan","BackendData":{"VtepMAC":"2a:2c:21:3a:58:21"}}
[root@mcwk8s03 mcwtest]# etcdctl get /coreos.com/network/subnets/172.17.83.0-24
{"PublicIP":"10.0.0.33","BackendType":"vxlan","BackendData":{"VtepMAC":"b2:83:33:7b:fd:37"}}
[root@mcwk8s03 mcwtest]#
【5】重啟005的flannel服務,如果不重啟docker0,那么網絡就會有點問題,docker0不會被分配新的網段IP
[root@mcwk8s05 ~]# systemctl restart flanneld.service
[root@mcwk8s05 ~]# ifconfig flannel.1
flannel.1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1450inet 172.17.89.0 netmask 255.255.255.255 broadcast 0.0.0.0inet6 fe80::3470:76ff:feea:39b8 prefixlen 64 scopeid 0x20<link>ether 36:70:76:ea:39:b8 txqueuelen 0 (Ethernet)RX packets 1 bytes 40 (40.0 B)RX errors 0 dropped 0 overruns 0 frame 0TX packets 9 bytes 540 (540.0 B)TX errors 0 dropped 8 overruns 0 carrier 0 collisions 0[root@mcwk8s05 ~]# ifconfig docker
docker0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1450inet 172.17.98.1 netmask 255.255.255.0 broadcast 172.17.98.255inet6 fe80::42:18ff:fee1:e8fc prefixlen 64 scopeid 0x20<link>ether 02:42:18:e1:e8:fc txqueuelen 0 (Ethernet)RX packets 551507 bytes 216568663 (206.5 MiB)RX errors 0 dropped 0 overruns 0 frame 0TX packets 635860 bytes 891305864 (850.0 MiB)TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0[root@mcwk8s05 ~]# systemctl restart docker
[root@mcwk8s05 ~]# ifconfig docker
docker0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1450inet 172.17.89.1 netmask 255.255.255.0 broadcast 172.17.89.255inet6 fe80::42:18ff:fee1:e8fc prefixlen 64 scopeid 0x20<link>ether 02:42:18:e1:e8:fc txqueuelen 0 (Ethernet)RX packets 552135 bytes 216658479 (206.6 MiB)RX errors 0 dropped 0 overruns 0 frame 0TX packets 636771 bytes 892057926 (850.7 MiB)TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0[root@mcwk8s05 ~]#
【6】再次查看,etcd已經有了05節點的這個網段了
[root@mcwk8s03 mcwtest]# etcdctl ls /coreos.com/network/subnets
/coreos.com/network/subnets/172.17.83.0-24
/coreos.com/network/subnets/172.17.9.0-24
/coreos.com/network/subnets/172.17.89.0-24
[root@mcwk8s03 mcwtest]# etcdctl get /coreos.com/network/subnets/172.17.89.0-24
{"PublicIP":"10.0.0.35","BackendType":"vxlan","BackendData":{"VtepMAC":"36:70:76:ea:39:b8"}}
[root@mcwk8s03 mcwtest]#
【7】再次測試,05節點的,可以nodeport訪問到了
[root@mcwk8s03 mcwtest]# kubectl get svc
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
kubernetes ClusterIP 10.2.0.1 <none> 443/TCP 583d
mcwtest-svc NodePort 10.2.0.155 <none> 2024:33958/TCP 154m
nginx ClusterIP None <none> 80/TCP 413d
[root@mcwk8s03 mcwtest]# curl -I 10.2.0.155:2024
curl: (7) Failed connect to 10.2.0.155:2024; Connection timed out
[root@mcwk8s03 mcwtest]# curl -I 10.0.0.33:33958
curl: (7) Failed connect to 10.0.0.33:33958; Connection refused
[root@mcwk8s03 mcwtest]# curl -I 10.0.0.35:33958
HTTP/1.0 200 OK
Server: SimpleHTTP/0.6 Python/2.7.18
Date: Tue, 04 Jun 2024 16:59:03 GMT
Content-type: text/html; charset=ANSI_X3.4-1968
Content-Length: 816[root@mcwk8s03 mcwtest]# curl -I 10.0.0.36:33958
HTTP/1.0 200 OK
Server: SimpleHTTP/0.6 Python/2.7.18
Date: Tue, 04 Jun 2024 16:59:12 GMT
Content-type: text/html; charset=ANSI_X3.4-1968
Content-Length: 816[root@mcwk8s03 mcwtest]#
【8】03節點不通,可能是03是master,但是它本身好像不作為node,就是個單純的master,所以沒法當做nodeIP去訪問。但是集群IP,無法訪問,是怎么回事呢
[root@mcwk8s03 mcwtest]# kubectl get svc
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
kubernetes ClusterIP 10.2.0.1 <none> 443/TCP 583d
mcwtest-svc NodePort 10.2.0.155 <none> 2024:33958/TCP 157m
nginx ClusterIP None <none> 80/TCP 413d
[root@mcwk8s03 mcwtest]# kubectl get nodes
NAME STATUS ROLES AGE VERSION
mcwk8s05 Ready <none> 580d v1.15.12
mcwk8s06 Ready <none> 580d v1.15.12
[root@mcwk8s03 mcwtest]#
【9】路由方面,重啟flannel,etcd里面會重新寫入新的網段,并且其他節點也會有這個新的網段的路由,需要重啟該宿主機的docker,給容器重新分配新的IP用吧,應該。
之前03 master只有一個flanel的路由,
[root@mcwk8s03 mcwtest]# route -n
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
0.0.0.0 10.0.0.254 0.0.0.0 UG 100 0 0 eth0
10.0.0.0 0.0.0.0 255.255.255.0 U 100 0 0 eth0
172.17.9.0 172.17.9.0 255.255.255.0 UG 0 0 0 flannel.1
172.17.83.0 0.0.0.0 255.255.255.0 U 0 0 0 docker0
[root@mcwk8s03 mcwtest]#05 node重啟之后,03上多出來一個05 node的flannel路由,
[root@mcwk8s03 mcwtest]# route -n
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
0.0.0.0 10.0.0.254 0.0.0.0 UG 100 0 0 eth0
10.0.0.0 0.0.0.0 255.255.255.0 U 100 0 0 eth0
172.17.9.0 172.17.9.0 255.255.255.0 UG 0 0 0 flannel.1
172.17.83.0 0.0.0.0 255.255.255.0 U 0 0 0 docker0
172.17.89.0 172.17.89.0 255.255.255.0 UG 0 0 0 flannel.1
[root@mcwk8s03 mcwtest]# 06好的node也是如此
之前:
[root@mcwk8s06 ~]# route -n
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
0.0.0.0 10.0.0.254 0.0.0.0 UG 100 0 0 eth0
10.0.0.0 0.0.0.0 255.255.255.0 U 100 0 0 eth0
172.17.9.0 0.0.0.0 255.255.255.0 U 0 0 0 docker0
172.17.83.0 172.17.83.0 255.255.255.0 UG 0 0 0 flannel.1
[root@mcwk8s06 ~]#
操作之后,多了個路由
[root@mcwk8s06 ~]# route -n
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
0.0.0.0 10.0.0.254 0.0.0.0 UG 100 0 0 eth0
10.0.0.0 0.0.0.0 255.255.255.0 U 100 0 0 eth0
172.17.9.0 0.0.0.0 255.255.255.0 U 0 0 0 docker0
172.17.83.0 172.17.83.0 255.255.255.0 UG 0 0 0 flannel.1
172.17.89.0 172.17.89.0 255.255.255.0 UG 0 0 0 flannel.1
[root@mcwk8s06 ~]# 05node之前是有兩個路由,但是都是錯誤的
[root@mcwk8s05 /]# route -n
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
0.0.0.0 10.0.0.254 0.0.0.0 UG 100 0 0 eth0
10.0.0.0 0.0.0.0 255.255.255.0 U 100 0 0 eth0
172.17.59.0 172.17.59.0 255.255.255.0 UG 0 0 0 flannel.1
172.17.61.0 172.17.61.0 255.255.255.0 UG 0 0 0 flannel.1
172.17.98.0 0.0.0.0 255.255.255.0 U 0 0 0 docker0
[root@mcwk8s05 /]#05重啟之后,也更新了路由為正確的
[root@mcwk8s05 /]# route -n
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
0.0.0.0 10.0.0.254 0.0.0.0 UG 100 0 0 eth0
10.0.0.0 0.0.0.0 255.255.255.0 U 100 0 0 eth0
172.17.9.0 172.17.9.0 255.255.255.0 UG 0 0 0 flannel.1
172.17.83.0 172.17.83.0 255.255.255.0 UG 0 0 0 flannel.1
172.17.89.0 0.0.0.0 255.255.255.0 U 0 0 0 docker0
[root@mcwk8s05 /]#
【10】然后看看集群IP為啥不通,這直接原因應該是沒有ipvs規則。
[root@mcwk8s03 mcwtest]# kubectl get svc
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
kubernetes ClusterIP 10.2.0.1 <none> 443/TCP 583d
mcwtest-svc NodePort 10.2.0.155 <none> 2024:33958/TCP 3h3m
nginx ClusterIP None <none> 80/TCP 413d
[root@mcwk8s03 mcwtest]# curl -I 10.2.0.155:2024
curl: (7) Failed connect to 10.2.0.155:2024; Connection timed out
[root@mcwk8s03 mcwtest]# ipvsadm -Ln
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags-> RemoteAddress:Port Forward Weight ActiveConn InActConn
[root@mcwk8s03 mcwtest]# systemctl status kube-proxy
Unit kube-proxy.service could not be found.
[root@mcwk8s03 mcwtest]# 其它節點都是有ipvs規則的,有時間確認下,ipvsadm規則是哪個服務創建的,應該是kube-proxy創建的吧,我們的03 master是沒有部署kube-proxy,這樣的話,沒有ipvs規則并且無法在master上訪問集群IP就說的通了。如過是這樣的話,也就是容器集群IP之間的通信,跟ipvs有關,跟apiserver是否掛了沒有直接關系,不影響,有時間驗證
[root@mcwk8s05 /]# ipvsadm -Ln|grep -C 1 10.2.0.155-> 172.17.89.4:9090 Masq 1 0 0
TCP 10.2.0.155:2024 rr-> 172.17.89.10:20000 Masq 1 0 0
[root@mcwk8s05 /]#
[root@mcwk8s05 /]# curl -I 10.2.0.155:2024
HTTP/1.0 200 OK
Server: SimpleHTTP/0.6 Python/2.7.18
Date: Tue, 04 Jun 2024 17:34:45 GMT
Content-type: text/html; charset=ANSI_X3.4-1968
Content-Length: 816[root@mcwk8s05 /]#
【綜上】
單節點的容器網絡用nodeIP:nodeport等之類的用不了了,可以優先檢查下docker0 網關是不是不通了;如果flannel等網絡服務重啟之后,即使是pod方式部署的網絡插件,也要看下重啟docker服務,讓容器分配新的網段。比如某次flannel oom總是重啟,導致kube001網絡故障;每個機器,好像都有其他節點的flannel.1網卡網段的路由,使用當前機器的flannel.1的網卡接口;并且有當前節點docker0網段的路由,是走的 0.0.0.0網關,走默認路由由上面可以知道,svc里面有個是clusterIP用的端口,clusterIP是用ipvs規則管理,進行數據轉發的;nodeip clusterIP的ipvs規則進行轉發,因為轉發到后端容器IP,當前宿主機有所有node的flannel網段的路由,所有flannel網關以及docker0都正常且對應網段的話,那么就是用這兩個接口實現容器跨宿主機通信的。而至于訪問nodeip:nodeport以及clusterIP:port是怎么知道把流量給到正確的pod的,這里是通過ipvs規則來實現尋找到后端pod的,轉發到pod對應的IP和端口的時候,又根據當前機器有所有網絡插件flannel,也就是每個宿主機單獨網段的路由條目,讓它們知道自己要去走flannel.1接口。而etcd保存有哪個網段是哪個宿主機,有對應宿主機IP,找到宿主機IP了,那么在該機器上轉發到的pod機器,在那個宿主機上就是能用該IP,使用docker0進行通信的,因為單個宿主機上的容器,都是通過docker0進行通信,并且互通的。只有跨宿主機通信容器的時候,才會根據路由,找到flannel.1接口,然后在etcd找到是那個宿主機上的容器,然后找到這個容器,完成通信。flannel網絡插件這樣,其它網絡插件原理類似;ipvs網絡代理模式是這個作用,iptables網絡代理模式作用類似。
文章轉載自:馬昌偉
原文鏈接:https://www.cnblogs.com/machangwei-8/p/18236417
體驗地址:引邁 - JNPF快速開發平臺_低代碼開發平臺_零代碼開發平臺_流程設計器_表單引擎_工作流引擎_軟件架構