方案
+---------------------+
| Jenkins Master |
| - 持久化配置 |<---(hostpath 存儲)
| - 自動容災 |
+----------+----------+|| Jenkins JNLP 通信|
+----------v----------+ +-------------------+
| Jenkins Agent | | Kubernetes Plugin |
| (動態 Pod) |<---->| (自動創建 Agent) |
+---------------------+ +-------------------+^|
+----------+----------+
| 彈性擴容機制 |
| (HPA + Pod 模板) |
+---------------------+
一、環境準備
k8s 集群ip
kubectl get nodes -o wide # 顯示各節點的 InternalIP 和 ExternalIP
檢查pod外網是否可通
# 創建臨時 Pod 測試公網訪問
kubectl run net-test --image=alpine --rm -it --restart=Never -- sh# 在 Pod 內部執行（測試 HTTP 訪問）
curl -v -4 --connect-timeout 10 http://example.com# 測試 DNS 解析（確認域名解析是否正常）
nslookup google.com# 測試 TCP 連接（如需要）
nc -zv www.baidu.com 80
jenkins鏡像：
docker login kubernetes-register.sswang.com:80 -u sswang
docker pull jenkins/jenkins:2.344 # 官方 2.34 最新版 :cite[2]:cite[5]:cite[7]
docker tag jenkins/jenkins:2.344 kubernetes-register.sswang.com:80/jenkins/jenkins:2.344
docker push kubernetes-register.sswang.com:80/jenkins/jenkins:2.344
docker pull jenkins/jenkins:lts-jdk17
docker tag jenkins/jenkins:lts-jdk17 kubernetes-register.sswang.com:80/jenkins/jenkins:lts-jdk17
docker push kubernetes-register.sswang.com:80/jenkins/jenkins:lts-jdk17
- agent鏡像
docker pull jenkins/inbound-agent:jdk11
docker tag jenkins/inbound-agent:jdk11 kubernetes-register.sswang.com:80/jenkins/inbound-agent:jdk11
docker push kubernetes-register.sswang.com:80/jenkins/inbound-agent:jdk11
- 創建namespace
kubectl create namespace jenkins
- hostpath存儲準備
kubectl label node kubernetes-master jenkins-master=true # node打標簽，支持基于標簽選擇器進行pod調度
mkdir -p /mnt/jenkins-data
chown 1000:1000 /mnt/jenkins-data # Jenkins 默認 UID 1000 deck1
chmod -R 755 /mnt/jenkins-data
- 版本兼容性說明
二、jenkins master部署
1. jenkins-master.yaml
apiVersion: apps/v1
kind: Deployment
metadata:name: jenkins-masternamespace: jenkins
spec:replicas: 1 # 副本數量selector:matchLabels:app: jenkins-masterstrategy:type: Recreatetemplate:metadata:labels:app: jenkins-masterspec:# 綁定到專用節點（可選）nodeSelector:jenkins-master: "true"securityContext:fsGroup: 1000containers:- name: jenkinsimage: kubernetes-register.sswang.com:80/jenkins/jenkins:lts-jdk17env:- name: JAVA_OPTSvalue: "-Djenkins.install.runSetupWizard=false -Dorg.apache.commons.jelly.tags.fmt.timeZone=Asia/Shanghai"ports:- containerPort: 8080- containerPort: 50000volumeMounts:- name: jenkins-homemountPath: /var/jenkins_homelivenessProbe:httpGet:path: /loginport: 8080initialDelaySeconds: 120periodSeconds: 10readinessProbe:httpGet:path: /loginport: 8080initialDelaySeconds: 30resources:limits:memory: "2Gi"cpu: "1"requests:memory: "1Gi"cpu: "500m"volumes:- name: jenkins-homehostPath:path: /mnt/jenkins-datatype: DirectoryOrCreate
2. jenkins-service.yml
nodeport 對外暴露jenkis服務，能夠訪問集群內pod。
apiVersion: v1
kind: Service
metadata:name: jenkins-servicenamespace: jenkins
spec:selector:app: jenkins-mastertype: NodePort # 永久對外暴露服務ports:- name: httpport: 8080targetPort: 8080- name: agentport: 50000targetPort: 50000
kubectl get svc -n jenkins jenkins-service
其中，32251為外部訪問所使用的端口，ip為宿主機ip或者虛擬機的ip地址。
Jenkins 服務地址確認
# 查看 Jenkins Service 名稱
kubectl get svc -n jenkins# 驗證內部 DNS 解析（在任意 Pod 內）
kubectl run test-dns --image=busybox -it --rm --restart=Never -- \nslookup jenkins-service.jenkins.svc.cluster.local# [svc].[namespace].svc.cluster.local
3. jenkins-rbac.yml
創建相應的RBAC權限，以便Jenkins Master能夠創建和管理Agent Pod
- ServiceAccount：在jenkins命名空間中創建一個名為jenkins的服務賬戶。
- ClusterRole：定義了一組權限規則，允許對pods、pods/exec、pods/log等資源的操作，以及獲取secrets的權限。
- ClusterRoleBinding：將ClusterRole綁定到ServiceAccount，使得jenkins服務賬戶具有ClusterRole中定義的權限。
apiVersion: v1
kind: ServiceAccount
metadata:name: jenkins # jenkins serviceaccountnamespace: jenkins
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:name: jenkins-agent-role
rules:
- apiGroups: [""]resources: ["pods", "pods/exec", "pods/log"]verbs: ["create", "delete", "get", "list", "watch", "patch"]
- apiGroups: [""]resources: ["secrets"]verbs: ["get"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:name: jenkins-agent-role-binding
roleRef:apiGroup: rbac.authorization.k8s.iokind: ClusterRolename: jenkins-agent-role
subjects:
- kind: ServiceAccountname: jenkinsnamespace: jenkins
kubectl get serviceaccount -n jenkins
如何讓某個訪問作為serviceaccount 為jenkins的賬戶？
apiVersion: v1
kind: Secret
metadata:name: jenkins-tokennamespace: jenkinsannotations:kubernetes.io/service-account.name: jenkins
type: kubernetes.io/service-account-token
至此就能通過secret訪問serviceaccount，進而訪問apiserver擁有相應的pod操作權限。
在這里插入圖片描述
三、動態Agent配置與kubernetes插件
1. 插件初始化
https://mirrors.huaweicloud.com/jenkins/updates/update-center.json
名稱: kubernetes
Kubernetes 地址: https://kubernetes.default.svc.cluster.local
命名空間: jenkins
憑據: jenkins-service-account (選擇創建的 ServiceAccount)
Jenkins 地址: http://jenkins-service.jenkins.svc.cluster.local:8080
憑據獲取，要求k8s集群配置：serviceaccount賬戶作為 集群操作權限的承接者，secret綁定account，用于apiserver鑒權，rbac用于將serviceaccount綁定role。
TOKEN=$(kubectl get secret jenkins-token -n jenkins -o jsonpath='{.data.token}' | base64 --decode)
echo $TOKEN
服務證書key獲取：
CA_CRT=$(kubectl get secret jenkins-token -n jenkins -o jsonpath='{.data.ca\.crt}' | base64 -d)
echo "$CA_CRT"
# 驗證 ServiceAccount 權限
kubectl auth can-i create pods \--as system:serviceaccount:jenkins:jenkins \-n jenkins
# 應返回 yes# 驗證列表權限
kubectl auth can-i list pods \--as system:serviceaccount:jenkins:jenkins \-n jenkins
# 應返回 yes
# [命名空間][serviceaccount]
2. pod 模板
名稱: jnlp-agent
標簽: jnlp-agent
容器模板:
- 名稱: jnlp鏡像: kubernetes-register.sswang.com:80/jenkins/inbound-agent:jdk11參數: '${computer.jnlpmac} ${computer.name}'資源限制:CPU: 500m內存: 512Mi資源請求:CPU: 200m內存: 256Mi
podTemplate:name: jenkins-agentlabel: jenkins-agentserviceAccount: jenkins
# imagePullSecrets:
# - name: docker-registry-credcontainers:- name: jnlpimage: kubernetes-register.sswang.com:80/jenkins/inbound-agent:jdk11alwaysPullImage: falseresourceLimitCpu: "1"resourceLimitMemory: "2Gi"args: '$(JENKINS_SECRET) $(JENKINS_NAME)'
)
)
