docker + K3S + Jenkins + Harbor自動化部署

最近公司在研究自動化部署的一套流程，下面記錄一下配置流程

需要提前準備好Jenkins? Harbor? Git(其他管理工具也可以)

我這里的打包編譯流程是Jenkins上配置打包任務-->自動到git目錄下找打包文件---->項目編譯后打鏡像包------>打完鏡像包將鏡像上傳到harbor倉庫----->通過yaml配置文件手動部署（最后一步其實也可以合并到Jenkins中，需要的自行學習）

一.準備打包文件

我的項目是springboot+vue 的所以我把打包文件分開寫了，你也可以合并到一起。

文件中標紅的內容需要替換自己的信息

1.Spring boot后臺項目目錄下添加打包文件

后臺Jenkinsfile文件內容

pipeline {
agent {
node {
label 'maven'
}

}

parameters {
gitParameter name: 'BRANCH_NAME', branch: '', branchFilter: '.*',defaultValue: 'master',description: '請選擇要發布的分支',quickFilterEnabled: false,type: 'PT_BRANCH_TAG'
choice(name: 'NAMESPACE', choices: ['master', 'fea-alert','dev-sit'],description: '命名空間')
string(name: 'TAG_NAME', defaultValue: 'snapshot', description: '標簽名稱，必須以 v 開頭，例如：v1、v1.0.0')
}
environment {
DOCKER_CREDENTIAL_ID = 'harbor-user-pass'?// 替換為你的 Docker Hub 憑證 ID
GIT_REPO_URL = '192.168.xxx.xxx' //git地址
GIT_CREDENTIAL_ID = 'private-git-token' //git憑證id
GIT_ACCOUNT = 'usernamexxx' //git賬號
REGISTRY = '192.168.xxx.xxx:80' //harbor地址
DOCKERHUB_NAMESPACE = 'zdyxxx' //harbor命名空間換自己的
APP_NAME = 'itwg-monitor-platformxxx' //應用名稱

}
stages {
stage('chechout scm') {
steps {
checkout scmGit(branches: [[name: "$BRANCH_NAME"]], extensions: [], userRemoteConfigs: [[credentialsId: "$GIT_CREDENTIAL_ID", url: "https://192.168.xxx.xxx/xxx.git"]])
}
}
stage('unit test manager') {
steps {
withMaven(jdk: 'jdk17') {
sh 'mvn clean test -f zdy-hertzbeat/pom.xml -P prod'
}
}
}
stage('build & push mamager') {
steps {
withMaven(jdk: 'jdk17') {
sh 'mvn clean package -DskipTests -f zdy-hertzbeat/pom.xml'
}
sh 'docker build -f zdy-hertzbeat/kubernetes/build/manage/Dockerfile -t $REGISTRY/$DOCKERHUB_NAMESPACE/$APP_NAME:SNAPSHOT-$BUILD_NUMBER .'
withCredentials([usernamePassword(passwordVariable : 'DOCKER_PASSWORD' ,usernameVariable : 'DOCKER_USERNAME' ,credentialsId : "$DOCKER_CREDENTIAL_ID")]) {
sh 'echo "$DOCKER_PASSWORD" | docker login $REGISTRY -u "$DOCKER_USERNAME" --password-stdin'
sh 'docker push $REGISTRY/$DOCKERHUB_NAMESPACE/$APP_NAME:SNAPSHOT-$BUILD_NUMBER'
}

}
}

stage('push latest manager') {
steps {
sh 'docker tag $REGISTRY/$DOCKERHUB_NAMESPACE/$APP_NAME:SNAPSHOT-$BUILD_NUMBER $REGISTRY/$DOCKERHUB_NAMESPACE/$APP_NAME:latest'
sh 'docker push $REGISTRY/$DOCKERHUB_NAMESPACE/$APP_NAME:latest'
}
}
}
}

后臺Dockerfile

## 基礎鏡像

FROM openjdk:17

## 作者
LABEL org.opencontainers.image.authors="738749649@qq.com"

## 定義參數

## 創建并進入工作目錄
#RUN mkdir -p /project/hlxx/config
#RUN mkdir -p /project/xcyth
WORKDIR /project/zrq
## maven 插件構建時得到 buildArgs 種的值
COPY zdy-hertzbeat/hertzbeat-manager/target/*.jar app.jar

## 設置 TZ 時區
## 設置 JAVA_OPTS 環境變量，可通過 docker run -e "JAVA_OPTS=" 進行覆蓋
ENV TZ=Asia/Shanghai JAVA_OPTS="-Xms2048m -Xmx4096m"

## 暴露端口
EXPOSE 1157

## 容器啟動命令
## CMD 第一個參數之后的命令可以在運行時被替換
CMD java ${JAVA_OPTS} -Djava.security.egd=file:/dev/./urandom -jar /project/zrq/app.jar

?2.vue前臺添加打包文件

前端Jenkinsfile內容如下：

pipeline {

agent {

node {

label 'maven'

}

}

parameters {

gitParameter name: 'BRANCH_NAME', branch: '', branchFilter: '.*',defaultValue: 'main',description: '請選擇要發布的分支',quickFilterEnabled: false,type: 'PT_BRANCH_TAG'

choice(name: 'NAMESPACE', choices: ['main'],description: '命名空間')

string(name: 'TAG_NAME', defaultValue: 'snapshot', description: '標簽名稱，必須以 v 開頭，例如：v1、v1.0.0')

}

environment {

DOCKER_CREDENTIAL_ID = 'harbor-user-pass'?// 替換為你的 Docker Hub 憑證 ID
GIT_REPO_URL = '192.168.xxx.xxx' //git地址
GIT_CREDENTIAL_ID = 'private-git-token' //git憑證id
GIT_ACCOUNT = 'usernamexxx' //git賬號
REGISTRY = '192.168.xxx.xxx:80' //harbor地址
DOCKERHUB_NAMESPACE = 'zdyxxx' //harbor命名空間換自己的

WEB_NAME = 'itwg-monitor-webxxx'

}

stages {

stage('chechout scm') {

steps {

checkout scmGit(branches: [[name: "$BRANCH_NAME"]], extensions: [], userRemoteConfigs: [[credentialsId: "$GIT_CREDENTIAL_ID", url: "https://192.168.xxx.xxx/xxxx.git"]])

}

}

//

stage('build & push') {

steps {

nodejs('nodejs20') {

dir('zdy-hertzbeat-ui-vue') {

sh 'pnpm install'

sh 'pnpm run build'

}

}

sh 'docker build -f kubernetes/build/web/Dockerfile -t $REGISTRY/$DOCKERHUB_NAMESPACE/$WEB_NAME:SNAPSHOT-$BUILD_NUMBER .'

withCredentials([usernamePassword(passwordVariable : 'DOCKER_PASSWORD' ,usernameVariable : 'DOCKER_USERNAME' ,credentialsId : "$DOCKER_CREDENTIAL_ID")]) {

sh 'echo "$DOCKER_PASSWORD" | docker2 login $REGISTRY -u "$DOCKER_USERNAME" --password-stdin'

sh 'docker push $REGISTRY/$DOCKERHUB_NAMESPACE/$WEB_NAME:SNAPSHOT-$BUILD_NUMBER'

}

}

}

stage('push latest') {

steps {

sh 'docker tag $REGISTRY/$DOCKERHUB_NAMESPACE/$WEB_NAME:SNAPSHOT-$BUILD_NUMBER $REGISTRY/$DOCKERHUB_NAMESPACE/$WEB_NAME:latest'

sh 'docker push $REGISTRY/$DOCKERHUB_NAMESPACE/$WEB_NAME:latest'

}

}

}

}

前端Dockerfile文件：

FROM docker.m.daocloud.io/nginx

LABEL org.opencontainers.image.authors="738749649@qq.com"

# 復制自定義的 Nginx 配置文件到鏡像中

COPY kubernetes/build/nginx/nginx.conf /etc/nginx/nginx.conf

ADD dist/ /usr/share/nginx/html/

EXPOSE 80

CMD ["/bin/sh", "-c", "nginx -g 'daemon off;'"]

?

前端nginx.conf

user nginx;

worker_processes 1;

error_log /var/log/nginx/error.log warn;

pid /var/run/nginx.pid;

events {

worker_connections 1024;

}

http {

include /etc/nginx/mime.types;

default_type application/octet-stream;

log_format main '$remote_addr - $remote_user [$time_local] "$request" '

'$status $body_bytes_sent "$http_referer" '

'"$http_user_agent" "$http_x_forwarded_for"';

access_log /var/log/nginx/access.log main;

sendfile on;

keepalive_timeout 65;

open_file_cache max=1000 inactive=20s;

open_file_cache_valid 30s;

open_file_cache_min_uses 2;

open_file_cache_errors on;

# client_max_body_size 用來修改允許客戶端上傳文件的大小。默認為1m，如果設置為0，表示上傳文件大小不受限制。

# 可以在以下模塊設置: http, server, location

client_max_body_size 100m;

# include /etc/nginx/conf.d/*.conf;

server {

listen 80 default_server;

listen [::]:80 default_server;

server_name localhost;

#

root /usr/share/nginx/html; # Vue 靜態文件根目錄（Docker 中默認掛載的 dist 目錄）

index index.html;

location / {

try_files $uri $uri/ /index.html; # 支持 Vue Router 的 history 模式

}

location /api {

proxy_set_header Host $host;

proxy_set_header X-Real-IP $remote_addr;

proxy_set_header REMOTE-HOST $remote_addr;

proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;

#itwg-monitor-platform kubectl啟動后臺的服務名:端口

proxy_pass http://itwg-monitor-platform:1157/api;

}

#

error_page 500 502 503 504 /50x.html;

location = /50x.html {

root /usr/share/nginx/html;

}

}

}

二.Jenkins新建任務?

上面我們準備好了我們打包需要的配置文件，這些文件都需要我們上傳項目git

到這里我們就可以在Jenkins上配置打包流程了。

我的項目是前后端分離的所以我配置兩個任務，一個后端，一個前端

下面截圖只是后端配置任務，前端的一樣只需修改對應的Jenkins腳本路徑和git配置

?三.Jenkins編譯打包

每個任務都需要編譯

四.在服務器部署

在服務器部署，需要我們寫個配置文件

配置文件中的image鏡像需要修改為自己的

前臺配置文件web.yaml

apiVersion: apps/v1
kind: Deployment
metadata:annotations:labels:app: itwg-monitor-webname: itwg-monitor-webnamespace: hertzbeat
spec:replicas: 1selector:matchLabels:app: itwg-monitor-webstrategy:rollingUpdate:maxSurge: 25%maxUnavailable: 25%type: RollingUpdatetemplate:metadata:labels:app: itwg-monitor-webspec:containers:- image: itwg-monitor-web  #鏡像imagePullPolicy: IfNotPresent   # 關鍵配置！name: itwg-monitor-webresources: {}terminationMessagePath: /dev/termination-logterminationMessagePolicy: FilednsPolicy: ClusterFirstrestartPolicy: AlwaysschedulerName: default-schedulersecurityContext: {}terminationGracePeriodSeconds: 30
----
apiVersion: v1
kind: Service
metadata:labels:app: itwg-monitor-webname: itwg-monitor-webnamespace: hertzbeat
spec:ports:- port: 80protocol: TCPtargetPort: 80nodePort: 30000selector:app: itwg-monitor-webtype: NodePort

后臺service.yaml

apiVersion: apps/v1
kind: Deployment
metadata:labels:app: itwg-monitor-platformname: itwg-monitor-platformnamespace: hertzbeat
spec:replicas: 1selector:matchLabels:app: itwg-monitor-platformstrategy:rollingUpdate:maxSurge: 25%maxUnavailable: 25%type: RollingUpdatetemplate:metadata:creationTimestamp: nulllabels:app: itwg-monitor-platformspec:containers:- name: itwg-monitor-platformimage: 192.168.6.114:80/zdy/itwg-monitor-platformimagePullPolicy: Alwaysresources: {}terminationMessagePath: /dev/termination-logterminationMessagePolicy: FilevolumeMounts:# 將宿主機的容器日志路徑掛載為 /usr/local/logs- name: container-logsmountPath: /usr/local/logsreadOnly: truevolumes:- name: container-logshostPath:path: /var/log/containerstype: DirectorydnsPolicy: ClusterFirstrestartPolicy: AlwaysschedulerName: default-schedulersecurityContext: {}terminationGracePeriodSeconds: 30
----
apiVersion: v1
kind: Service
metadata:labels:app: itwg-monitor-platformname: itwg-monitor-platformnamespace: hertzbeat
spec:ports:- port: 1157protocol: TCPtargetPort: 1157selector:app: itwg-monitor-platformtype: ClusterIP

?部署命令：

kubectl apply -f??service.yaml? -n? 命名空間

kubectl apply -f??web.yaml? -n? 命名空間

查看命令

kubectl get po -n??命名空間

kubectl get svc -n?命名空間

kubectl 常用命令

查看日志

kubectl describe pod <pod-name> -n <namespace>

kubectl logs <pod-name> -n <namespace>

刪除

kubectl delete svc hertzbeat-manager -n hertzbeat

kubectl delete deploy hertzbeat-manager -n hertzbeat

?五.問題記錄

? 1.如果Jenkins編譯前端無法install，編譯報錯提示

ERR_PNPM_OUTDATED_LOCKFILE ?Cannot install with "frozen-lockfile" because pnpm-lock.yaml is not up ?

執行pnpm install --no-frozen-lockfile

命令，然后將pnpm-lock.yaml文件提交后再進行編譯

? 2.如果前臺后端都部署到服務器前臺無法到服務器

先檢查前臺打包配置文件env.production中配置地址是否正確

檢查掛載到nginx中的代理轉發是否正確

3.如果遇到Maven打包后打包鏡像運行找不到主類問題，需要在maven打包中添加下面配置

<plugin><groupId>org.springframework.boot</groupId><artifactId>spring-boot-maven-plugin</artifactId><!--加入下面兩項配置--><executions><execution><goals><goal>repackage</goal></goals></execution></executions><configuration><includeSystemScope>true</includeSystemScope></configuration>
</plugin><!-- Maven Dependency Plugin -->
<plugin><groupId>org.apache.maven.plugins</groupId><artifactId>maven-dependency-plugin</artifactId><version>3.7.0</version><executions><execution><id>copy-dependencies</id><phase>package</phase><goals><goal>copy-dependencies</goal></goals><configuration><outputDirectory>${project.build.directory}/lib</outputDirectory></configuration></execution></executions>
</plugin>