Gitlab-Runner安裝

文章目錄

helm方式安裝在K8S上
參考
- gitlab CI/CD 文件變量
- 緩存服務器
- - K8S部署
- docker鏡像
- - maven
  - docker
  - - 安裝docker buildx
  - minio
  - node
  - helm
  - kubectl
  - sonar-scanner-cli
- 問題
- - 清除cache
  - helm執行時無權限
- 下載鏡像失敗
- - 下載gitlab-runner鏡像失敗
- Gitlab-ci中使用
- - java
  - 前端

helm方式安裝在K8S上

1、下載charts

helm pull gitlab/gitlab-runner
tar -zxvf gitlab-runner-0.27.0.tgz#解壓后內容：CHANGELOG.mdChart.yaml    #CONTRIBUTING.mdLICENSEMakefileNOTICEREADME.mdtemplates    #values.yaml  #

2、修改 values.yaml，templates 等資源

values.yaml

gitlabUrl: https://gitlab.example.com/  #修改為gitlab地址
runnerRegistrationToken: ""             #修改為gitlab runner token，可從 /admin/runners 查看
rbac:create: trueclusterWideAccess: trueserviceAccountName: gitlab-runner-gitlab-runner
runners:tags: ""  serviceAccountName: gitlab-runner-gitlab-runner

templates/configmap.yaml

主要用于maven，docker綁定本地目錄，修改 entrypoint key。增加 config.toml 配置。

    #以下一段是增加的內容cat >>/home/gitlab-runner/.gitlab-runner/config.toml <<EOF[[runners.kubernetes.volumes.host_path]]name = "maven"mount_path = "/root/.m2"read_only = falsehost_path = "/root/.m2"[[runners.kubernetes.volumes.host_path]]name = "docker"mount_path = "/var/run/docker.sock"read_only = truehost_path = "/var/run/docker.sock"EOF# Start the runnerexec /entrypoint run --user=gitlab-runner \--working-directory=/home/gitlab-runner

新的方式可以通過values.yaml 的 runners 段設置屬性。不能同時以上面和下面2種方式，不然會重復。

runners:config: |[[runners]][runners.kubernetes]image = "ubuntu:16.04"[[runners.kubernetes.volumes.host_path]]name = "maven"mount_path = "/root/.m2"read_only = falsehost_path = "/root/.m2"[[runners.kubernetes.volumes.host_path]]name = "docker"mount_path = "/var/run/docker.sock"read_only = truehost_path = "/var/run/docker.sock"

_cache.tpl

里面CACHE_S3_INSECURE 參數是固定值，導致 values 配置無效。

{{-       if .Values.runners.cache.s3CacheInsecure }}
- name: CACHE_S3_INSECUREvalue: "true"
{{-       end }}{{ default "" .Values.runners.cache.s3BucketLocation | quote }}#-----   修改為：- name: CACHE_S3_INSECUREvalue: {{ default "true" .Values.runners.cache.s3CacheInsecure | quote }}

3、添加 helm 倉庫

helm repo add gitlab https://charts.gitlab.io

4、創建namespace、等資源

kubectl create ns gitlab

---
apiVersion: v1
data:accesskey: bWluaW8=     #base64 編碼secretkey:     #base64 編碼
kind: Secret
metadata:name: minio-secrets
type: Opaque

5、啟動 gitlab-runner

# 安裝倉庫中的chart
$ helm install   gitlab-runner   --namespace gitlab    -f values.yaml gitlab/gitlab-runner  
#安裝本地的chart
helm install   gitlab-runner  ./   --namespace gitlab#更新配置--通過本地chart更新helm upgrade --install   gitlab-runner    ./gitlab-runner  --namespace gitlab #卸載
helm uninstall gitlab-runner --namespace gitlab

如果沒有修改gitlabUrl，則會提示更新配置

#############################################################################################
## WARNING: You did not specify an gitlabUrl in your 'helm install' call.                  ##
#############################################################################################This deployment will be incomplete until you provide the URL that your
GitLab instance is reachable at:helm upgrade gitlab-runner \--set gitlabUrl=http://gitlab.your-domain.com,runnerRegistrationToken=your-registration-token \gitlab/gitlab-runner#也可以使用命令：helm upgrade

參考

安裝：https://docs.gitlab.com/runner/install/

https://docs.gitlab.com/runner/

執行器參數：https://docs.gitlab.com/runner/executors/kubernetes.html

cache secret ： https://blog.csdn.net/xichenguan/article/details/101436883

gitlab runner配置（toml配置項）：https://docs.gitlab.com/runner/configuration/advanced-configuration.html

gitlab CI/CD 文件變量

新的版本支持，比較舊的不支持。

但是可以通過base64 編解碼來實現

echo $(cat ~/.kube/config | base64) | tr -d " "

deploy_k8s_job:image: registry.cn-hangzhou.aliyuncs.com/haoshuwei24/kubectl:1.16.6stage: deploy_k8stags:- k8s-runnerscript:- mkdir -p /etc/deploy- echo $kube_config |base64 -d > $KUBECONFIG- sed -i "s/IMAGE_TAG/$CI_PIPELINE_ID/g" deployment.yaml- cat deployment.yaml- kubectl apply -f deployment.yaml

緩存服務器

使用minio作為緩存服務器。配置如下：

  cache:## General settings## DEPRECATED: See https://docs.gitlab.com/runner/install/kubernetes.html#additional-configuration and https://docs.gitlab.com/runner/install/kubernetes.html#using-cache-with-configuration-templatecacheType: s3cachePath: "gitlab_runner"cacheShared: true## S3 settings## DEPRECATED: See https://docs.gitlab.com/runner/install/kubernetes.html#additional-configuration and https://docs.gitlab.com/runner/install/kubernetes.html#using-cache-with-configuration-templates3ServerAddress: s3.amazonaws.coms3BucketName: "gitlabrunner"          #Minio buckets3BucketLocation:        #minio時區。s3CacheInsecure: false   #是否在不安全模式。true：使用http；false使用https，不設置則默認為false。## S3 the name of the secret.secretName: minio-secrets     #minio 對應的secret

**注意：**很多博客或者什么資料，把s3CacheInsecure解釋為是否使用https，正確的解釋應該是是否在不安全模式。意思剛好相反。

最終的文件內容可以在/home/gitlabrunner/.gitlabrunner/config.toml 文件查看。值為false時不會出現在config.toml中。

以上方式是廢棄的方式，新的方式采用template。對應的template為_cache.yaml

runners:config: |[[runners]][runners.kubernetes]image = "ubuntu:16.04"[runners.cache]Type = "s3"Path = "gitlab_runner"Shared = true[runners.cache.s3]ServerAddress = "s3.amazonaws.com"BucketName = "gitlabrunner"BucketLocation = "eu-west-1"Insecure = true#AccessKey = "access"   #SecretKey = "secret123456"cache:secretName: minio-secrets

以上使用到了一個secret。通過以下語句創建secret 或者通過yaml創建。

kubectl create secret generic minio \
--from-literal=accesskey="access" \
--from-literal=secretkey="secret123456" -n gitlab

參考：https://docs.gitlab.com/runner/install/kubernetes.html#using-cache-with-configuration-template

K8S部署

---
kind: PersistentVolumeClaim
apiVersion: v1
metadata:name: miniofinalizers:- kubernetes.io/pvc-protection
spec:accessModes:- ReadWriteOnceresources:requests:storage: 5GistorageClassName: rook-cephfsvolumeMode: Filesystem---
apiVersion: v1
kind: Service
metadata:labels:app: minioname: miniospec:ports:- name: 9000-tcpport: 9000protocol: TCPtargetPort: 9000selector:app: miniosessionAffinity: Nonetype: ClusterIP---
apiVersion: apps/v1
kind: Deployment
metadata:labels:app: minioname: miniospec:replicas: 1revisionHistoryLimit: 10selector:matchLabels:app: miniotemplate:metadata:labels:app: miniospec:containers:- image: minio/minio:RELEASE.2019-02-26T19-51-46ZimagePullPolicy: Alwaysenv:- name: MINIO_ACCESS_KEYvalue: minio- name: MINIO_SECRET_KEY  value: sssscommand:- minio- server- /dataname: minioports:- containerPort: 9000protocol: TCPterminationMessagePath: /dev/termination-logterminationMessagePolicy: FilevolumeMounts:- mountPath: /dataname: volume-datadnsPolicy: ClusterFirstrestartPolicy: AlwaysschedulerName: default-schedulerimagePullSecrets:- name: harbor-key   #注意docker 倉庫 keysecurityContext: {}terminationGracePeriodSeconds: 30volumes:- name: volume-datapersistentVolumeClaim:claimName: minio

kubectl apply -f minio.single.yaml -n gitlab

docker鏡像

maven

maven:3.6.3-openjdk-8： https://registry.hub.docker.com/_/maven

maven的setting.xml 可以通過configmap解決，（沒驗證過）

        [[runners.kubernetes.volumes.config_map]]name = "gitlab-runner-maven"mount_path = "/usr/share/maven/configmap/"

也可以通過mount path解決（見前面內容）

[[runners.kubernetes.volumes.host_path]]

docker

docker ：https://registry.hub.docker.com/_/docker 。版本：（20.10.2）

需要在/root/.docker/config.json 中增加auth 憑據。

FROM docker
MAINTAINER lihz
ADD  config.json  /root/.docker/config.json

config.json

主要是增加訪問憑據

{"auths": {"192.168.1.X": {"auth": "?????????????"},"docker-registry-default.cloud.com": {"auth": "YWRtaW46TEpWUUhYX2g3MGFabGYtUlJLdDc1RlBmRW5LeFRXXXXXXXXXXX"}},"experimental": true
}

安裝docker buildx

如果需要支持多平臺打包，則需要安裝docker buildx (github.com/docker/buildx v0.10.5 86bdced)，下載

wget -O docker-buildx  https://github.com/docker/buildx/releases/download/v0.10.5/buildx-v0.10.5.linux-amd64
mkdir -p /usr/libexec/docker/cli-plugins/docker-buildx
mv docker-buildx /usr/libexec/docker/cli-plugins/docker-buildx
chmod +x /usr/libexec/docker/cli-plugins/docker-buildxdocker buildx version

docker客戶端需要開啟實驗室功能

$ cat ~/.docker/config.json
{"experimental": "enabled"
}# 確認實驗室性能開啟。
$ docker version

構造docker 打包的鏡像，包含buildx

FROM docker:20.10.2
MAINTAINER lihz
ADD  config.json  /root/.docker/config.json
RUN mkdir -p /usr/libexec/docker/cli-plugins/  && mkdir -p /etc/docker
COPY docker-buildx /usr/libexec/docker/cli-plugins/docker-buildx
COPY daemon.json buildkitd.toml  /etc/docker/
RUN chmod +x /usr/libexec/docker/cli-plugins/docker-buildx
ENV IMAGE_BUILDKIT=192.168.1.X/GROUP/buildkit:buildx-stable-1

buildkitd.toml

debug = true
# insecure-entitlements allows insecure entitlements, disabled by default.
insecure-entitlements = [ "network.host", "security.insecure" ]# 如果不加這些，就會默認使用https請求。
# optionally mirror configuration can be done by defining it as a registry.
[registry."192.168.1.XX"]http = trueinsecure = true

打包

minio

minio/minio:RELEASE.2019-02-26T19-51-46Z : https://registry.hub.docker.com/r/minio/minio

node

node:14.7.0 : https://registry.hub.docker.com/_/node

FROM node:14.7.0
RUN npm config set registry https://registry.npm.taobao.org

helm

alpine/helm:3.5.0：https://registry.hub.docker.com/r/alpine/helm

Dockerfile：

From 192.168.1.X/GROUP/helm:3.5.0
#增加K8S的憑據
ADD config /etc/deploy/config

config：

K8S的憑據

apiVersion: v1
clusters:
- cluster:certificate-authority-data: ........server: https://lb.kubesphere.local:6443name: cluster.local
contexts:
- context:cluster: cluster.localnamespace: demouser: kubernetes-adminname: ctx-demo
- context:cluster: cluster.localuser: kubernetes-adminname: kubernetes-admin@cluster.local
current-context: ctx-demo
kind: Config
preferences: {}
users:
- name: kubernetes-adminuser:client-certificate-data: ..........client-key-data: ..........

kubectl

將業務鏡像部署到k8s上

sonar-scanner-cli

用于掃描前端代碼。參考：https://docs.sonarqube.org/latest/analysis/scan/sonarscanner/

sonarsource/sonar-scanner-cli:4.6：https://registry.hub.docker.com/r/sonarsource/sonar-scanner-cli

Dockerfile：

From sonarsource/sonar-scanner-cli:4.6
#登錄憑據
ENV SONAR_HOST_URL=http://192.168.1.XXX:9000 SONAR_LOGIN=a34d8e475e19faa108404fec82cd058493XXXXXX
ENTRYPOINT [""]

綁定目錄：

docker run --rm -v $PWD:/usr/src

問題

https://docs.gitlab.com/ee/ci/docker/using_docker_build.html

清除cache

cache是沒有過期時間的，而且每一次新的push觸發的pipeline，都會重新生成cache，重新生成的cache的名字為“-”，其中num是隨著push數量遞增的。如果不去清除cache，cache會永久保留在Runner上，日積月累會填滿存儲空間的，因此最好隔一段時間進行一次清除，清除方法請參考https://docs.gitlab.com/ee/ci/caching/#clearing-the-cache，或者使用clear_volumes.sh 這個簡單腳本來處理它, 清除cache的原理是將相關的volume移除，當然，docker也有自帶的清除命令，推薦將docker system prune -f --volumes加入到定時任務中。

helm執行時無權限

Executing "step_script" stage of the job script
$ sed -i "s/IMAGE_TAG/$DOCKER_TAG/g;s/CI_PROJECT_NAME/$CI_PROJECT_NAME/g;s/SVC_PORT/${SVC_PORT}/g;" ${MODULE_NAME}/src/main/charts/values.yaml
$ sed -i "s/CI_PROJECT_NAME/$CI_PROJECT_NAME/g" ${MODULE_NAME}/src/main/charts/Chart.yaml
$ helm upgrade --install $CI_PROJECT_NAME ${MODULE_NAME}/src/main/charts -n $K8S_NS
Release "sample" does not exist. Installing it now.
Error: rendered manifests contain a resource that already exists. Unable to continue with install: could not get information about the resource: deployments.apps "sample" is forbidden: User "system:serviceaccount:gitlab:gitlab-runner-gitlab-runner" cannot get resource "deployments" in API group "apps" in the namespace "release"
ERROR: Job failed: command terminated with exit code 1

是由于 gitlab runner的權限問題

執行以下語句：

kubectl create clusterrolebinding gitlab-cluster-admin --clusterrole=cluster-admin --group=system:serviceaccounts

下載鏡像失敗

Job failed (system failure): prepare environment: image pull failed

臨時解決方法，在K8S節點 docker pull <IMAGE> 把鏡像下載下來

根本性解決：

打開以下選項，并設置docker倉庫的secret。

## Specifying ImagePullSecrets on a Pod （設置在gitlab-runner中）
## Kubernetes supports specifying container image registry keys on a Pod.
## ref: https://kubernetes.io/docs/concepts/containers/images/#specifying-imagepullsecrets-on-a-pod
##
imagePullSecrets:- name: "harbor-key"## For RBAC support:
rbac:create: true## Specify one or more imagePullSecrets used for pulling the runner image#### ref: https://kubernetes.io/docs/tasks/configure-pod-container/configure-service-account/#add-imagepullsecrets-to-a-service-account##imagePullSecrets: ["harbor-key"]## Configuration for the Pods that the runner launches for each new job
##
runners:## Specify one or more imagePullSecrets  （用于拉取image）#### ref: https://kubernetes.io/docs/tasks/configure-pod-container/pull-image-private-registry/#### DEPRECATED: See https://docs.gitlab.com/runner/install/kubernetes.html#additional-configurationimagePullSecrets: ["harbor-key"]## Run all containers with the privileged flag enabled## This will allow the docker:dind image to run if you need to run Docker## commands. Please read the docs before turning this on:## ref: https://docs.gitlab.com/runner/executors/kubernetes.html#using-dockerdind#### DEPRECATED: See https://docs.gitlab.com/runner/install/kubernetes.html#additional-configuration#privileged: true

下載gitlab-runner鏡像失敗

在K8S部署環境中，會下載以下鏡像，可能會導致失敗，最好重新tag在本地倉庫

# helm配置（helpers.tpl中）為：
printf "192.168.1.X/GROUP/gitlab-runner:alpine-%s" $appVersion
#tag為：
192.168.1.x/GROUP/gitlab-runner:alpine-v13.8.0# 最后一部分是 CI_RUNNER_VERSION，對應的版本的 sha256ID，參考：https://gitlab.com/gitlab-org/gitlab-runner/-/tags?sort=updated_desc&search=13.8.0
gitlab/gitlab-runner-helper:x86_64-775dd39d

docker tag gitlab/gitlab-runner-helper:x86_64-775dd39d   192.168.1.X/GROUP/gitlab-runner-helper:x86_64-775dd39d
docker push 192.168.1.X/GROUP/gitlab-runner-helper:x86_64-775dd39d

修改配置：

      [[runners]][runners.kubernetes]image = "ubuntu:22.04"# 由上文可知helper_image = "192.168.1.X/GROUP/gitlab-runner-helper:x86_64-775dd39d"

Gitlab-ci中使用

java

variables:DOCKER_TAG: "3.0.0-RELEASE"MODULE_NAME: "project-biz"SONAR_PROJECT_KEY: "project"stages:- package- docker_buildmvn_build_job:image: ${DEPOSITORY}/mavenstage: packagescript:- mvn clean verify sonar:sonar -DskipTests -DskipDocker -Dsonar.projectVersion=master -Dsonar.projectKey=$SONAR_PROJECT_KEY  -Dsonar.host.url=${SONAR_URL} -Dsonar.login=${SONAR_TOKEN}- mvn deploy -B -DskipTests -DskipDockerartifacts:paths:- ${MODULE_NAME}/target/*.jaronly:- master- /^.*-dev$/when: manualmvn_build_release_job:image: ${DEPOSITORY}/mavenstage: packagescript:- mvn deploy -B -DskipTests -DskipDockerartifacts:paths:- ${MODULE_NAME}/target/*.jaronly:- /^.*-RELEASE$/- /^.*-release/- /^.*-hotfix$/docker_build_release_job:image: ${DEPOSITORY}/dockerstage: docker_buildscript:- cp ${MODULE_NAME}/target/*.jar ${MODULE_NAME}/src/main/docker- docker build -t ${DEPOSITORY}/${MODULE_NAME}:${DOCKER_TAG} ${MODULE_NAME}/src/main/docker- docker push ${DEPOSITORY}/${MODULE_NAME}:${DOCKER_TAG}only:- /^.*-RELEASE$/- /^.*-release/

前端

variables:DOCKER_TAG: "dev"MODULE_NAME: "biz-web"stages:- package- docker_build- deploy npm_build_job:image: maven:3.6.3-openjdk-8stage: packagecache:paths:- node_modules/artifacts:paths:- distscript:- npm install- npm run buildonly:- master- /^.*-dev$/when: manual  docker_build_job:image: dockerstage: docker_buildscript:- docker build -t ${DEPOSITORY}/${MODULE_NAME}:${DOCKER_TAG} ./dependencies:- npm_build_jobonly:- master- /^.*-dev$/when: manualdocker_build_release_job:image: dockerstage: docker_buildscript:- docker build -t ${DEPOSITORY}/${MODULE_NAME}:${DOCKER_TAG} ./dependencies:- npm_build_job  only:- /^.*-RELEASE$/- /^.*-release/- /^.*-hotfix$/