高可用架構

本文采用kubeadm方式搭建k8s高可用集群，k8s高可用集群主要是對apiserver、etcd、controller-manager、scheduler做的高可用；高可用形式只要是為：

1. apiserver利用haproxy+keepalived做的負載，多apiserver節點同時工作；

2. etcd利用k8s內部提供的解決方案做的高可用，最多容忍一臺etcd服務宕機

3. controller-manager、scheduler的高可用機制為如果多個服務同時存在，則會選舉一個leader工作，另外兩個處于sleep的狀態

集群信息規劃

主機名稱	系統版本	主機IP	備注
K8s-master01	Centos7.6	192.168.0.101	Master01/registry
K8s-master02	Centos7.6	192.168.0.102	Master02
K8s-master03	Centos7.6	192.168.0.103	Master03
K8s-node01	Centos7.6	192.168.0.111	Node01
K8s-node02	Centos7.6	192.168.0.112	Node02
K8s-node03	Centos7.6	192.168.0.113	Node03
Apiserver-keepalived	Centos7.6	192.168.0.100	HA-apiserver

服務器的初始化

#此些操作根據實際需求所有服務器都需要操作?

修改hosts文件?

192.168.0.101 k8s-master01 192.168.0.102 k8s-master02? 192.168.0.103 k8s-master03 192.168.0.111 k8s-node01 192.168.0.112 k8s-node02 192.168.0.113 k8s-node03

修改主機名稱

hostnamectl set-hostname k8s-master01 hostname -b k8s-master01 #其余服務器按照此方式進行修改

關閉NetworkManager服務

[root@k8s-master01 ~]# systemctl stop NetworkManager
[root@k8s-master01 ~]# systemctl disable NetworkManager
Removed symlink /etc/systemd/system/multi-user.target.wants/NetworkManager.service.
Removed symlink /etc/systemd/system/dbus-org.freedesktop.nm-dispatcher.service.
Removed symlink /etc/systemd/system/network-online.target.wants/NetworkManager-wait-online.service.

修改服務器的IP地址

[root@k8s-master01 ~]# sed -i '/IP/d' /etc/sysconfig/network-scripts/ifcfg-ens33
[root@k8s-master01 ~]# sed -i 's/BOOTPROTO=dhcp/BOOTPROTO=static/g' /etc/sysconfig/network-scripts/ifcfg-ens33
[root@k8s-master01 ~]# sed -i 's/ONBOOT=no/ONBOOT=yes/g' /etc/sysconfig/network-scripts/ifcfg-ens33
[root@k8s-master01 ~]# sed -i '/UUID/d' /etc/sysconfig/network-scripts/ifcfg-ens33
[root@k8s-master01 ~]# echo -e "IPADDR=192.168.0.101\nNETMASK=255.255.255.0\nGATEWAY=192.168.0.1" >> /etc/sysconfig/network-scripts/ifcfg-ens33
[root@k8s-master01 ~]# systemctl restart network

安裝依賴包

[root@k8s-master01 ~]# yum install -y conntrack ntpdate ntp ipvsadm ipset jq iptables curl sysstat libseccomp wgetvimnet-tools git

設置防火墻并且清空規則

[root@k8s-master01 ~]# systemctl stop firewalld && systemctl disable firewalld
[root@k8s-master01 ~]#yum -y install iptables-services && systemctl start iptables && systemctl enable iptables&& iptables -F && service iptables save

關閉selinux

[root@k8s-master02 ~]# swapoff -a && sed -i '/ swap / s/^\(.*\)$/#\1/g' /etc/fstab
[root@k8s-master01 ~]# setenforce 0 && sed -i 's/^SELINUX=.*/SELINUX=disabled/' /etc/selinux/config

服務器之間免密配置

#生成密鑰
[root@k8s-master01 ~]# ssh-keygen -t rsa#將密鑰傳到其他服務器上面
[root@k8s-master01 ~]# ssh-copy-id -i /root/.ssh/id_rsa.pub root@192.168.0.102

Docker依賴安裝

[root@k8s-master01 ~]# yum install -y   yum-utils device-mapper-persistent-data lvm2

安裝Docker源

[root@k8s-master01 ~]# yum-config-manager --add-repo  http://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo

安裝Docker

[root@k8s-master01 ~]# yum update -y && yum install -y docker-ce

配置daemon配置文件

[root@k8s-master01 ~]# mkdir -p /etc/docker
[root@k8s-master01 ~]# mkdir -p  /etc/systemd/system/docker.service.d
[root@k8s-master01 ~]# cat > /etc/docker/daemon.json < {
> "exec-opts": ["native.cgroupdriver=systemd"], #著重注意標紅字體
> "log-driver": "json-file",
> "log-opts": {
> "max-size": "100m"
> },
> "insecure-registries":["http://registry.k8s-test.com"] #后面會使用到私有鏡像倉庫
> }
> EOF

啟動docker并且設置開機自啟動

[root@k8s-master01 ~]# systemctl daemon-reload && systemctl restart docker && systemctl enable docker

安裝kubelet

[root@k8s-master01 ~]# cat  <  /etc/yum.repos.d/kubernetes.repo
> [kubernetes]
> name=Kubernetes
> baseurl=http://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64
> enabled=1
> gpgcheck=0
> repo_gpgcheck=0
> gpgkey=http://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg
> http://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg
> EOF[root@k8s-master01 ~]# yum install -y kubelet-1.16.4 kubeadm-1.16.4 kubectl-1.16.4

啟動kubelet并且設置為開機啟動

[root@k8s-master01 ~]#  systemctl enable kubelet && systemctl start kubelet

Keepalive安裝

[root@k8s-master01 ~]# yum -y install keepalived

修改配置文件:

master01和master02和master03配置一樣，只是router_id記得變一下

[root@k8s-master01 ~]# more /etc/keepa hived/keepalhived.Conf
! Configuration File for keepalived 
global_ defs{router_id master01
}
vrrp_instance VI_1 {state BACKUPinterface ens33virtual_router_id 5priority 90advert_int 1authentication {auth type PASSauth pass 1111}virtual ipaddress {192.168.0.100}
}

啟動keepalived

[root@k8s-master02 ~]# systemctl enable keepalived && systemctl start keepalived

驗證

master01

master02

master03

k8s安裝

下載鏡像

[root@k8s-master01 tools]# sh get_images.sh

部署鏡像中心

[root@k8s-master01 images]# docker images | grep k8s-registry.com/registry[root@k8s-master01 images]# docker run -d -p 80:5000 -v /home/registry:/var/lib/registry --restart=always --name registry k8s-registry.com/registry:1.0

將鏡像push到鏡像中心

[root@k8s-master01 images]# docker images | awk '{print $1":"$2}' | xargs -i docker push {}

創建kubeadm-config.yaml的配置文件

[root@k8s-master01 install-master]# more  kubeadm-config.yaml
apiVersion: kubeadm.k8s.io/v1beta2
kind: ClusterConfiguration
kubernetesVersion: v1.16.4
apiServer:certSANs:- k8s-master01- k8s-master02- k8s-master03- k8s-node01- k8s-node02- k8s-node03- 192.168.0.100- 192.168.0.101- 192.168.0.102- 192.168.0.103- 192.168.0.111- 192.168.0.112- 192.168.0.113
controlPlaneEndpoint: "192.168.0.100:6443"
networking:podSubnet: "10.244.0.0/16"

初始化master

[root@k8s-master01 install-master]# kubeadm init --config=kubeadm-config.yaml

驗證安裝

創建flannel網絡

下載flannel的yaml文件

[root@k8s-master01 flannel]# wget https://raw.githubusercontent.com/coreos/flannel/2140ac876ef134e0ed5af15c65e414cf26827915/Documentation/kube-flannel.yml

創建flannel網絡

[root@k8s-master01 flannel]# kubectl app -f kube-flannel.yml

驗證安裝

master節點添加至集群

master01向其他兩個節點頒發證書

[root@k8s-master01 tools]# cat cert-amin-master.sh
[root@k8s-master01 tools]# sh cert-amin-master.sh

其他節點將證書放到指定目錄

[root@k8s-master02 ~]# sh cert-other-master.sh
[root@k8s-master02 ~]# more cert-other-master.sh

加入集群

kubeadm join 192.168.0.100:6443 --token lllil4.2wm1u6ocuxmysn7l \--discovery-token-ca-cert-hash sha256:fa5075ba896b8dbfdaf19125dee28817fdd349b7c4cea9ab243ad4224eb90892 \--control-plane

查看布置的節點