kubernetes 集群部署

kubernetes 集群部署

環境
JiaoJiao_Centos7-1(152.112) 192.168.152.112
JiaoJiao_Centos7-2(152.113) 192.168.152.113
JiaoJiao_Centos7-3(152.114) 192.168.152.114
已開通 4C+8G+80G

集群規劃

?

部署方式

環境準備：基于主機名稱通信，時間同步，關閉firewall和iptables.service

方式一：yum ,rpm 安裝。復雜。

1. etcd cluster ：安裝在master節點上2. flannel：安裝在所有節點上
2. 安裝配置master
kube-apiserver,kube-scheduler,kube-controller-manager
3. 安裝配置node
kube-proxy
kubelet

方式二：使用kubeadm工具安裝。簡單。

1. master和node 都用yum 安裝kubelet，kubeadm，docker
2. master 上初始化：kubeadm init
3. master 上啟動一個flannel的pod
4. node上加入集群:kubeadm join

部署步驟

采取方式二：使用kubeadm工具安裝步驟

準備環境

修改主機名（3臺機器都需要修改）

# hostnamectl set-hostname master

配置hosts和dns解析（3臺機器都需要配置）

[root@node02 ~]# cat /etc/hosts
127.0.0.1 node02 localhost localhost.localdomain localhost4 localhost4.localdomain4
::1 node02 localhost localhost.localdomain localhost6 localhost6.localdomain6
192.168.152.112 master
192.168.152.113 node01
192.168.152.114 node02

默認的dns服務器沒法解析到opsx.alibaba.com和mirrors.aliyun.com，使用nameserver 192.168.?.?（具體根據你所在公司環境設置）這個可以。

[root@master ~]# cat /etc/resolv.conf 
# Generated by NetworkManager
nameserver 192.168.?.?

?

關閉防火墻和iptables，檢查iptables配置值是否為1（3臺機器都需要）

# systemctl stop firewalld.service
# systemctl stop iptables.service 
# systemctl disable firewalld.service
[root@master docker]# cat /proc/sys/net/bridge/bridge-nf-call-ip6tables 
1
[root@master docker]# cat /proc/sys/net/bridge/bridge-nf-call-iptables 
1

配置docker yum源

#yum update
#yum install -y yum-utils device-mapper-persistent-data lvm2 wget
#cd /etc/yum.repos.d
#wget https://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo

配置kubernetes yum 源

#wget https://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg
# wget https://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg
# rpm --import rpm-package-key.gpg
# rpm --import yum-key.gpg
[root@master yum.repos.d]# cat kubernetes.repo 
[kubernetes]
name=Kubernetes Repo
baseurl=https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64/
gpgcheck=1
gpgkey=https://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg
enabled=1
# yum repolist

master和node 安裝kubelet，kubeadm，docker

# yum install docker-ce kubelet kubeadm

master 上安裝kubectl ，node節點可以選擇安裝或者不安裝

kubectl 是API server 客戶端程序，通過連接master端的API server來實現k8s對象資源的增刪改查等基本操作。

#yum install kubectl

?

docker的配置

配置私有倉庫和鏡像加速地址

[root@master docker]# cat /etc/docker/daemon.json 
{"registry-mirrors": ["http://295c6a59.m.daocloud.io"],
"insecure-registries":["自己搭建的私有倉庫地址"]}

?

設置代理

[root@master docker]# cat /usr/lib/systemd/system/docker.service |grep -v "^#"
[Unit]
Description=Docker Application Container Engine
Documentation=https://docs.docker.com
BindsTo=containerd.service
After=network-online.target firewalld.service containerd.service
Wants=network-online.target
Requires=docker.socket
[Service]
Type=notify
ExecStart=/usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock
ExecReload=/bin/kill -s HUP $MAINPID
TimeoutSec=0
RestartSec=2
Restart=always
#設置代理地址，和不讓代理的地址
Environment="HTTPS_PROXY=http://www.ik8s.io:10080" "NO_PROXY=localhost,127.0.0.1,192.168.152.0/24"
StartLimitBurst=3
StartLimitInterval=60s
LimitNOFILE=infinity
LimitNPROC=infinity
LimitCORE=infinity
TasksMax=infinity
Delegate=yes
KillMode=process
[Install]
WantedBy=multi-user.target

?

啟動docker

# systemctl daemon-reload 
# systemctl start docker 
[root@master docker]# docker info 
Containers: 0
Running: 0
Paused: 0
Stopped: 0
Images: 0
Server Version: 18.09.6
Storage Driver: overlay2
Backing Filesystem: xfs
Supports d_type: true
Native Overlay Diff: false
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
Volume: local
Network: bridge host macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: bb71b10fd8f58240ca47fbb579b9d1028eea7c84
runc version: 2b18fe1d885ee5083ef9f0838fee39b62d653e30
init version: fec3683
Security Options:
seccomp
Profile: default
Kernel Version: 3.10.0-514.el7.x86_64
Operating System: CentOS Linux 7 (Core)
OSType: linux
Architecture: x86_64
CPUs: 4
Total Memory: 7.64GiB
Name: master
ID: KEBQ:4XXU:LTA7:J5QH:HK4F:JS26:RKI4:YFPJ:RY45:Q647:SJI6:VZDQ
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
HTTPS Proxy: http://www.ik8s.io:10080 #代理地址
No Proxy: localhost,127.0.0.1,192.168.152.0/24
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
XXXXX #本公司私有庫地址
127.0.0.0/8
Registry Mirrors:
http://295c6a59.m.daocloud.io/ #鏡像加速地址
Live Restore Enabled: false
Product License: Community Engine 
# systemctl enable docker

?

master 上初始化：kubeadm init

修改配置文件，忽略swap報錯

[root@master docker]# cat /etc/sysconfig/kubelet 
KUBELET_EXTRA_ARGS="--fail-swap-on=false"

?

初始化

#kubeadm init [command] --help 可以先查看下幫助
# kubeadm init --kubernetes-version=v1.14.2 --pod-network-cidr=10.244.0.0/16 --service-cidr=10.96.0.0/12 --ignore-preflight-errors=Swap--kubernetes-version=v1.14 # k8s的版本
版本通過：https://github.com/kubernetes/kubernetes/releases 來確定你需要的版本。
--pod-network-cidr=10.244.0.0/16 #pod網絡。flannel 的默認是10.244.0.0/16
--service-cidr=10.96.0.0/12 # service 網絡，也是用了默認值
--ignore-preflight-errors=Swap #忽略Swap報錯[root@master ~]# kubeadm init --kubernetes-version=v1.14.2 --pod-network-cidr=10.244.0.0/16 --service-cidr=10.96.0.0/12 --ignore-preflight-errors=Swap
[init] Using Kubernetes version: v1.14.2
[preflight] Running pre-flight checks
[WARNING IsDockerSystemdCheck]: detected "cgroupfs" as the Docker cgroup driver. The recommended driver is "systemd". Please follow the guide at https://kubernetes.io/docs/setup/cri/
[WARNING Swap]: running with swap on is not supported. Please disable swap
[WARNING Service-Kubelet]: kubelet service is not enabled, please run 'systemctl enable kubelet.service'
[preflight] Pulling images required for setting up a Kubernetes cluster
[preflight] This might take a minute or two, depending on the speed of your internet connection
[preflight] You can also perform this action in beforehand using 'kubeadm config images pull'
error execution phase preflight: [preflight] Some fatal errors occurred:
[ERROR ImagePull]: failed to pull image k8s.gcr.io/kube-apiserver:v1.14.2: output: Error response from daemon: Get https://k8s.gcr.io/v2/: proxyconnect tcp: dial tcp 172.96.236.117:10080: connect: connection refused
, error: exit status 1
[ERROR ImagePull]: failed to pull image k8s.gcr.io/kube-controller-manager:v1.14.2: output: Error response from daemon: Get https://k8s.gcr.io/v2/: proxyconnect tcp: dial tcp 172.96.236.117:10080: connect: connection refused
, error: exit status 1
[ERROR ImagePull]: failed to pull image k8s.gcr.io/kube-scheduler:v1.14.2: output: Error response from daemon: Get https://k8s.gcr.io/v2/: proxyconnect tcp: dial tcp 172.96.236.117:10080: connect: connection refused
, error: exit status 1

?

報錯了。拉不到鏡像。
解決辦法：docker.io倉庫對google的容器做了鏡像，可以通過下列命令下拉取相關鏡像

#docker pull mirrorgooglecontainers/kube-apiserver:v1.14.2
#docker pull mirrorgooglecontainers/kube-controller-manager:v1.14.2
#docker pull mirrorgooglecontainers/kube-scheduler:v1.14.2
#docker pull mirrorgooglecontainers/kube-proxy:v1.14.2
#docker pull mirrorgooglecontainers/pause:3.1
#docker pull mirrorgooglecontainers/etcd:3.3.10
#docker pull coredns/coredns:1.3.1

通過docker tag命令來修改鏡像的標簽：

# docker tag mirrorgooglecontainers/kube-apiserver:v1.14.2 k8s.gcr.io/kube-apiserver:v1.14.2
# docker tag mirrorgooglecontainers/kube-controller-manager:v1.14.2 k8s.gcr.io/kube-controller-manager:v1.14.2
# docker tag mirrorgooglecontainers/kube-scheduler:v1.14.2 k8s.gcr.io/kube-scheduler:v1.14.2 
# docker tag mirrorgooglecontainers/kube-proxy:v1.14.2 k8s.gcr.io/kube-proxy:v1.14.2 
# docker tag mirrorgooglecontainers/pause:3.1 k8s.gcr.io/pause:3.1 
# docker tag mirrorgooglecontainers/etcd:3.3.10 k8s.gcr.io/etcd:3.3.10
# docker tag coredns/coredns:1.3.1 k8s.gcr.io/coredns:1.3.1 

初始化過程中有提示，請按照提示進行操作并且將下面內容保存下來。

kubeadm join 192.168.152.112:6443 --token qlu35g.rfxl6k8mvry8nxfp \
> --discovery-token-ca-cert-hash sha256:8febc1b1402b471187e55186b4c3937ac23cfe830fb6ccfbd2e9f212e011218d

部署網絡：master 上啟動一個flannel的pod

下載鏡像。由于不能直接下載官方鏡像，自己百度找了個地址：

#docker pull jmgao1983/flannel:v0.11.0-amd64
#docker tag jmgao1983/flannel:v0.11.0-amd64 quay.io/coreos/flannel:v0.11.0-amd64
#docker rmi jmgao1983/flannel:v0.11.0-amd64

啟動：

# kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml

檢查下：

[root@master ~]# kubectl get pods -n kube-system
NAME READY STATUS RESTARTS AGE
coredns-fb8b8dccf-f7db6 1/1 Running 0 53m
coredns-fb8b8dccf-sd8hk 1/1 Running 0 53m
etcd-master 1/1 Running 0 52m
kube-apiserver-master 1/1 Running 0 52m
kube-controller-manager-master 1/1 Running 0 52m
kube-flannel-ds-amd64-zq5zr 1/1 Running 0 19m
kube-proxy-8mtbc 1/1 Running 0 53m
kube-scheduler-master 1/1 Running 0 52m

到此，master 就安裝好了!?保存鏡像

docker save -o master.tar k8s.gcr.io/kube-proxy:v1.14.2 k8s.gcr.io/kube-apiserver:v1.14.2 k8s.gcr.io/kube-controller-manager:v1.14.2 k8s.gcr.io/kube-scheduler:v1.14.2 quay.io/coreos/flannel:v0.11.0-amd64 k8s.gcr.io/coredns:1.3.1 k8s.gcr.io/etcd:3.3.10 k8s.gcr.io/pause:3.1

?

node節點加入集群

將master上保存的鏡像同步到節點上

[root@master ~]#scp master.tar node01:/root/
[root@master ~]#scp master.tar node02:/root/

將鏡像導入本地

[root@node01 ~]# docker load< master.tar

修改配置，忽略報錯：

[root@node01 docker]# cat /etc/sysconfig/kubelet 
KUBELET_EXTRA_ARGS="--fail-swap-on=false"

執行加入集群命令

[root@node01 ~]# kubeadm join 192.168.152.112:6443 --token qlu35g.rfxl6k8mvry8nxfp --discovery-token-ca-cert-hash sha256:8febc1b1402b471187e55186b4c3937ac23cfe830fb6ccfbd2e9f212e011218d --ignore-preflight-errors=Swap

在master上查看節點情況

[root@master ~]# kubectl get nodes
NAME STATUS ROLES AGE VERSION
master Ready master 107m v1.14.2
node01 Ready <none> 23m v1.14.2
node02 Ready <none> 23m v1.14.2

?