CICD-Devops整合Kubernetes-4

Devops整合Kubernetes

Kubernetes部署

快速安裝Kubernetes

**官網：**https://kuboard.cn/

選擇默認支持docker的版本1.19

前置環境部署

所有節點均需執行同操作

# 配置主機名解析
[root@Kubernetes-master ~]# echo "127.0.0.1 $(hostname)" >> /etc/hosts# 防火墻關閉，禁用selinux
[root@master ~]# systemctl disable firewalld.service --now
[root@master ~]# setenforce 0# 配置固定IP
[root@master ~]# vim /etc/sysconfig/network-scripts/ifcfg-ens33
TYPE="Ethernet"
PROXY_METHOD="none"
BROWSER_ONLY="no"
BOOTPROTO="static"
DEFROUTE="yes"
IPV4_FAILURE_FATAL="no"
IPV6INIT="yes"
IPV6_AUTOCONF="yes"
IPV6_DEFROUTE="yes"
IPV6_FAILURE_FATAL="no"
IPV6_ADDR_GEN_MODE="stable-privacy"
NAME="ens33"
UUID="f0967cdb-cfb4-4ed8-973b-03a171214395"
DEVICE="ens33"
ONBOOT="yes"
IPADDR="10.1.8.103"
NETMASK="255.255.255.0"
GATEWAY="10.1.8.2"
DNS1="223.5.5.5"
DNS1="223.6.6.6"
[root@Kubernetes-master ~]# systemctl restart network

安裝Docker和Kubelet

# 最后一個參數 1.19.5 用于指定 kubenetes 版本，支持所有 1.19.x 版本的安裝
# 騰訊云 docker hub 鏡像
# export REGISTRY_MIRROR="https://mirror.ccs.tencentyun.com"
# DaoCloud 鏡像
# export REGISTRY_MIRROR="http://f1361db2.m.daocloud.io"
# 華為云鏡像
# export REGISTRY_MIRROR="https://05f073ad3c0010ea0f4bc00b7105ec20.mirror.swr.myhuaweicloud.com"
# 阿里云 docker hub 鏡像
# export REGISTRY_MIRROR=https://registry.cn-hangzhou.aliyuncs.com
curl -sSL https://kuboard.cn/install-script/v1.19.x/install_kubelet.sh | sh -s 1.19.5

所有節點安裝

[root@Kubernetes-master ~]# export REGISTRY_MIRROR=https://registry.cn-hangzhou.aliyuncs.com
[root@Kubernetes-master ~]# curl -sSL https://kuboard.cn/install-script/v1.19.x/install_kubelet.sh | sh -s 1.19.5

初始化master節點

在master節點上進行初始化

修改：export MASTER_IP=10.1.8.103

export APISERVER_NAME=abner.com

其他內容保持不變進行復制粘貼

# 只在 master 節點執行
# 替換 x.x.x.x 為 master 節點實際 IP（請使用內網 IP）
# export 命令只在當前 shell 會話中有效，開啟新的 shell 窗口后，如果要繼續安裝過程，請重新執行此處的 export 命令
export MASTER_IP=10.1.8.103
# 替換 apiserver.demo 為 您想要的 dnsName
export APISERVER_NAME=abner.com
# Kubernetes 容器組所在的網段，該網段安裝完成后，由 kubernetes 創建，事先并不存在于您的物理網絡中
export POD_SUBNET=10.100.0.1/16
echo "${MASTER_IP} ${APISERVER_NAME}" >> /etc/hosts
curl -sSL https://kuboard.cn/install-script/v1.19.x/init_master.sh | sh -s 1.19.5

[root@Kubernetes-master ~]# export MASTER_IP=10.1.8.103
[root@Kubernetes-master ~]# export APISERVER_NAME=abner.com
[root@Kubernetes-master ~]# export POD_SUBNET=10.100.0.1/16
[root@Kubernetes-master ~]# echo "${MASTER_IP} ${APISERVER_NAME}" >> /etc/hosts
[root@Kubernetes-master ~]# curl -sSL https://kuboard.cn/install-script/v1.19.x/init_master.sh | sh -s 1.19.5

查看node狀態

# 查看結果等待全部running
[root@Kubernetes-master ~]# kubectl get pod -n kube-system -o wide
NAME                                        READY   STATUS    RESTARTS   AGE   IP             NODE                NOMINATED NODE   READINESS GATES
calico-kube-controllers-6c89d944d5-4tbwx    1/1     Running   1          60m   10.100.237.5   kubernetes-master   <none>           <none>
calico-node-cjnvr                           1/1     Running   1          60m   10.1.8.103     kubernetes-master   <none>           <none>
coredns-59c898cd69-99sqk                    1/1     Running   1          60m   10.100.237.6   kubernetes-master   <none>           <none>
coredns-59c898cd69-mrpjx                    1/1     Running   1          60m   10.100.237.4   kubernetes-master   <none>           <none>
etcd-kubernetes-master                      1/1     Running   2          61m   10.1.8.103     kubernetes-master   <none>           <none>
kube-apiserver-kubernetes-master            1/1     Running   2          61m   10.1.8.103     kubernetes-master   <none>           <none>
kube-controller-manager-kubernetes-master   1/1     Running   1          61m   10.1.8.103     kubernetes-master   <none>           <none>
kube-proxy-22tb9                            1/1     Running   2          60m   10.1.8.103     kubernetes-master   <none>           <none>
kube-scheduler-kubernetes-master            1/1     Running   1          61m   10.1.8.103     kubernetes-master   <none>           <none>
# 查看node狀態
[root@Kubernetes-master ~]# kubectl  get nodes -o wide
NAME                STATUS   ROLES    AGE   VERSION   INTERNAL-IP   EXTERNAL-IP   OS-IMAGE                KERNEL-VERSION          CONTAINER-RUNTIME
kubernetes-master   Ready    master   60m   v1.19.5   10.1.8.103    <none>        CentOS Linux 7 (Core)   3.10.0-693.el7.x86_64   docker://19.3.11

初始化node節點

獲取join命令參數

在master節點執行，生成的token有效時間為2小時

[root@Kubernetes-master ~]# kubeadm token create --print-join-command
W0814 10:55:55.430938   81817 configset.go:348] WARNING: kubeadm cannot validate component configs for API groups [kubelet.config.k8s.io kubeproxy.config.k8s.io]
kubeadm join abner.com:6443 --token vbxsvs.ijri7kwh4fdajzra     --discovery-token-ca-cert-hash sha256:467eedcc799fd0cd8fbe333c41fde8524373866df460d87b2e455134b5c50054

所有node節點執行

# 只在 node 節點執行
# 替換 x.x.x.x 為 master 節點的內網 IP
export MASTER_IP=10.1.8.103
# 替換 apiserver.demo 為初始化 master 節點時所使用的 APISERVER_NAME
export APISERVER_NAME=abner.com
echo "${MASTER_IP} ${APISERVER_NAME}" >> /etc/hosts
kubeadm join abner.com:6443 --token vbxsvs.ijri7kwh4fdajzra     --discovery-token-ca-cert-hash sha256:467eedcc799fd0cd8fbe333c41fde8524373866df460d87b2e455134b5c50054

[root@Kubernetes-node ~]# export MASTER_IP=10.1.8.103
[root@Kubernetes-node ~]# export APISERVER_NAME=abner.com
[root@Kubernetes-node ~]# echo "${MASTER_IP} ${APISERVER_NAME}" >> /etc/hosts
[root@Kubernetes-node ~]# kubeadm join abner.com:6443 --token vbxsvs.ijri7kwh4fdajzra     --discovery-token-ca-cert-hash sha256:467eedcc799fd0cd8fbe333c41fde8524373866df460d87b2e455134b5c50054

檢查初始化結果，等待一會，node節點為Ready狀態即可

[root@Kubernetes-master ~]# kubectl get node
NAME                STATUS     ROLES    AGE   VERSION
kubernetes-master   Ready      master   65m   v1.19.5
kubernetes-node     Ready      <none>   55s   v1.19.5

整合Yaml文件

因為yaml資源文件可以在k8s上創建應用pod，所以需要提前創建yaml，拉取到k8s服務器上在gitlab上新建YAML文件

gitlab頁面-項目中

apiVersion: apps/v1
kind: Deployment
metadata:name: pipelinelabels:app: pipeline
spec:replicas: 2selector:matchLabels:app: pipelinetemplate:metadata:labels:app: pipelinespec:containers:- name: pipelineimage: 10.1.8.102:80/repo/pipeline:v3.0imagePullPolicy: Always ports:- containerPort: 8080
---
apiVersion: v1
kind: Service
metadata:labels:app: pipelinename: pipeline
spec:selector:app: pipelineports:- port: 8081targetPort: 8080type: NodePort

對接harbor倉庫

在所有節點添加harbor倉庫地址配置

[root@Kubernetes-master ~]# cat /etc/docker/daemon.json 
{"insecure-registries": ["10.1.8.102:80"],  # 添加harbor倉庫"registry-mirrors": ["https://09def58152000fc00ff0c00057bad7e0.mirror.swr.myhuaweicloud.com"],"exec-opts": ["native.cgroupdriver=systemd"],"log-driver": "json-file","log-opts": {"max-size": "100m"},"storage-driver": "overlay2","storage-opts": ["overlay2.override_kernel_check=true"]
}
[root@Kubernetes-master ~]# systemctl restart docker# 測試 harbor節點登錄
[root@Kubernetes-master ~]# docker login -u admin -p Gzq20000308. 10.1.8.102:80 
WARNING! Using --password via the CLI is insecure. Use --password-stdin.
WARNING! Your password will be stored unencrypted in /root/.docker/config.json.
Configure a credential helper to remove this warning. See
https://docs.docker.com/engine/reference/commandline/login/#credentials-storeLogin Succeeded

YAML推送K8S服務器設置

主節點創建推送目標目錄

# 創建推送目標目錄
[root@Kubernetes-master ~]# mkdir /usr/local/k8s
[root@Kubernetes-master ~]# chmod 777 /usr/local/k8s
[root@Kubernetes-master ~]# ll /usr/local/ | grep k8s
drwxrwxrwx  2 root root  6 8月  14 11:32 k8s

jenkins-系統管理-系統配置

jenkins頁面-pipline項目-流水線語法

將流水線腳本復制到gitlab的Jenkinsfile中

sshPublisher(publishers: [sshPublisherDesc(configName: 'K8S', transfers: [sshTransfer(cleanRemote: false, excludes: '', execCommand: '', execTimeout: 120000, flatten: false, makeEmptyDirs: false, noDefaultExcludes: false, patternSeparator: '[, ]+', remoteDirectory: '', remoteDirectorySDF: false, removePrefix: '', sourceFiles: 'pipeline.yaml')], usePromotionTimestamp: false, useWorkspaceInPromotion: false, verbose: false)])

新增tag的標簽

構建測試yaml文件的推送

master節點上查看是否傳遞成功

[root@Kubernetes-master ~]# ls /usr/local/k8s/
pipeline.yaml

Jenkins使用無密碼登錄K8S的master節點

因為希望使用ssh 用戶名@k8s地址 kubectl apply -f /usr/local/k8s/pipeline.yml創建資源，會進行ssh免交互

把Jenkins中的公鑰內容傳遞給k8s的master實現免密

[root@Jenkins ~]# docker exec -it jenkins bash
jenkins@392ae884a4ea:/$ cd ~
jenkins@392ae884a4ea:~$ ssh-keygen -t rsa
Generating public/private rsa key pair.
Enter file in which to save the key (/var/jenkins_home/.ssh/id_rsa): 
Created directory '/var/jenkins_home/.ssh'.
Enter passphrase (empty for no passphrase): 
Enter same passphrase again: 
Your identification has been saved in /var/jenkins_home/.ssh/id_rsa
Your public key has been saved in /var/jenkins_home/.ssh/id_rsa.pub
The key fingerprint is:
SHA256:Xz49X2FABJOHBr9xElJvnMnHRJvIcpGriYsZaBSvSzg jenkins@392ae884a4ea
The key's randomart image is:
+---[RSA 3072]----+
|         oo==+oo |
|     .    o+OoB o|
|      o   .=.&.= |
|     . .    O.o  |
|    o o S ..+  o |
|   E = . o = .. .|
|    + . + o o o .|
|     . o .   . o.|
|                .|
+----[SHA256]-----+
jenkins@392ae884a4ea:~$ cd .ssh/
# 復制公鑰內容傳遞給master
jenkins@392ae884a4ea:~/.ssh$ cat id_rsa.pub 
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCyMzwHnfEvSdbgwFqlPPTqLZGCsDHoYkbwywAKsniM2nKfRXl3WCSgGvNF8RL9YJvcsmJSDuPtuXBX/8LlwT7Wf28pSApHuRq4ud9JwuAYd90vKUcopXAnWaxRHEWLMvXOg3H5w5pEgzk/KpJA39awA0TKaohc4nt2o+xvgXaCgHS3rMR75KV0ypS+2ABHEugK0hdHXx1lXNJzAn2L/wKB78DJXpmNv4pebW1x50uBZY8YZuShBo/0nzFajQzOW3G6zYYVma7wzUH3UU7aGeyN/goRVVk0EzVYEsRD0CLrr3I3WBlpFnD3TbVN6/70uuA2jjiHXCwpCCssVWsOQyXlw2GYABpmhgy0C5ekJrwdbsqI230JYO9VD1BrxAnEBlAUJtQfBIiQGrxYrSFEWHXtek86MVgRYetQHM/8VvuV+2MoGPYzZZgFNoARUMYocSJJU8YIsHjEF4xD7A6wKzlnMcr5eS/a56lx1MLdgrY0eAX7m1m2pD4IwDxJZrbzmMU= jenkins@392ae884a4ea

master節點

[root@Kubernetes-master ~]# mkdir .ssh
[root@Kubernetes-master ~]# cd .ssh/
[root@Kubernetes-master .ssh]# cat authorized_keys 
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCyMzwHnfEvSdbgwFqlPPTqLZGCsDHoYkbwywAKsniM2nKfRXl3WCSgGvNF8RL9YJvcsmJSDuPtuXBX/8LlwT7Wf28pSApHuRq4ud9JwuAYd90vKUcopXAnWaxRHEWLMvXOg3H5w5pEgzk/KpJA39awA0TKaohc4nt2o+xvgXaCgHS3rMR75KV0ypS+2ABHEugK0hdHXx1lXNJzAn2L/wKB78DJXpmNv4pebW1x50uBZY8YZuShBo/0nzFajQzOW3G6zYYVma7wzUH3UU7aGeyN/goRVVk0EzVYEsRD0CLrr3I3WBlpFnD3TbVN6/70uuA2jjiHXCwpCCssVWsOQyXlw2GYABpmhgy0C5ekJrwdbsqI230JYO9VD1BrxAnEBlAUJtQfBIiQGrxYrSFEWHXtek86MVgRYetQHM/8VvuV+2MoGPYzZZgFNoARUMYocSJJU8YIsHjEF4xD7A6wKzlnMcr5eS/a56lx1MLdgrY0eAX7m1m2pD4IwDxJZrbzmMU= jenkins@392ae884a4ea
# 重啟sshd服務
[root@Kubernetes-master ~]# systemctl restart sshd

Jenkins容器中驗證

jenkins@392ae884a4ea:~/.ssh$ ssh root@10.1.8.103 ls /
bin
boot
dev
dvd
etc
home
lib
lib64
media
mnt
opt
proc
root
run
sbin
srv
sys
tmp
usr
var

Jenkins中設置YAML部署

Jenkins頁面-pipline-流水線-流水線語法

sh 'ssh root@10.1.8.103 kubectl apply -f /usr/local/k8s/pipeline.yaml'

gitlab的jenkinsfile中增加步驟

修改pipeline.yaml拉取4.0鏡像

修改首頁內容

路徑：src/main/java/com/chenyan/demo/controller/TestController.java

gitlab中把原來的v4.0標簽刪除，重新創建，命名依然寫v4.0

Jenkins頁面-pipline-進行構建

# master節點上查看資源
[root@Kubernetes-master ~]# kubectl get pod -o wide
NAME                        READY   STATUS    RESTARTS   AGE   IP             NODE              NOMINATED NODE   READINESS GATES
pipeline-54f899865d-7gqgk   1/1     Running   0          12m   10.100.7.130   kubernetes-node   <none>           <none>
pipeline-54f899865d-8z6d4   1/1     Running   0          12m   10.100.7.129   kubernetes-node   <none>           <none>[root@Kubernetes-master ~]# kubectl get svc
NAME         TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)          AGE
kubernetes   ClusterIP   10.96.0.1       <none>        443/TCP          4h53m
pipeline     NodePort    10.96.166.168   <none>        8081:30831/TCP   10m# node上查看pipeline鏡像
[root@Kubernetes-node ~]# docker images 10.1.8.102:80/repo/pipeline
REPOSITORY                    TAG                 IMAGE ID            CREATED             SIZE
10.1.8.102:80/repo/pipeline   v4.0                99a6a3f548d5        13 minutes ago      543MB