如何添加極狐GitLab Runner 信任域名證書

本文作者徐曉偉

極狐Gitlab Runner 信任實例域名證書，用于注冊注冊極狐 GitLab Runner。

問題

參見極狐gitlab-runner-host.md

說明

解決方案是使用頒發給域名 gitlab.test.helm.xuxiaowei.cn 的證書，可以使用自己的域名去各大云廠商免費申請，或者使用自己根據域名 gitlab.test.helm.xuxiaowei.cn 生成的證書
1. 阿里云SSL(https)證書免費申請
2. 騰訊云SSL(https)證書免費申請
3. 華為云SSL(https)證書免費申請
4. 百度云SSL(https)證書免費申請

域名證書解決方案

方案1：重新配置極狐 GitLab，自動生成對應證書并自動配置，然后在極狐 GitLab runner 中信任證書

問題1：tls: failed to verify certificate: x509: certificate signed by unknown authority

文檔
1. runners 部分
2. 自簽名證書或自定義證書頒發機構
3. ConfigMap
4. 配置 Pod 使用 ConfigMap
根據上述文檔可知，有三種方式可以解決證書信任問題：
1. 在極狐GitLab Runner 注冊時配置 --tls-ca-file 參數并指向證書文件
2. 在配置文件中增加 tls-ca-file 并指向配置文件， 此方式本文不做介紹，可參考 gitlab-runner-kubernetes-host.md ，原理相同
3. 將證書放在 /etc/gitlab-runner/certs/ 文件夾下，使用域名命名證書，證書后綴名是 .crt， 由于篇幅限制，這里只介紹這一種方式
由于 k8s 的 pod 可能隨時會被刪除（故障轉移、主動刪除等），所以直接在 pod 內部配置不是上策
k8s 中的 ConfigMap 可以掛載到 pod 內部，所以可以考慮在 ConfigMap 中配置證書，然后 pod 內部使用 ConfigMap 中的證書

下載證書

方式1
1. 瀏覽器訪問域名 https://gitlab.test.helm.xuxiaowei.cn
2. 按照圖中操作，下載證書，上傳之服務器，用于后期操作

方式2

# 下載證書
openssl s_client -showcerts -connect gitlab.test.helm.xuxiaowei.cn:443 -servername gitlab.test.helm.xuxiaowei.cn < /dev/null 2>/dev/null | openssl x509 -outform PEM > gitlab.test.helm.xuxiaowei.cn.crt

將證書導入到 k8s 中

# -n=gitlab-test：指定命名空間
# create configmap etc-gitlab-runner-certs：創建 ConfigMap 名稱是 etc-gitlab-runner-certs
# --from-file=gitlab.test.helm.xuxiaowei.cn.crt=gitlab.test.helm.xuxiaowei.cn.crt：配置來自文件，文件名 gitlab.test.helm.xuxiaowei.cn.crt，放入 ConfigMap 中的鍵也是 gitlab.test.helm.xuxiaowei.cn.crt
kubectl -n=gitlab-test create configmap etc-gitlab-runner-certs --from-file=gitlab.test.helm.xuxiaowei.cn.crt=gitlab.test.helm.xuxiaowei.cn.crt# 查看
# kubectl -n=gitlab-test get configmap etc-gitlab-runner-certs -o yaml

導出 helm 極狐GitLab 配置

helm -n gitlab-test get values my-gitlab > my-gitlab.yaml

修改 helm 極狐GitLab 配置

gitlab-runner:# 掛載卷名稱volumeMounts:# 掛載到 pod 路徑（文件/文件夾）# 此處是在容器內運行的 gitlab runner，由于權限限制等原因，# 所以 配置文件不是在 /etc/gitlab-runner/ 目錄下，而是 /home/gitlab-runner/.gitlab-runner/- mountPath: /home/gitlab-runner/.gitlab-runner/certsname: etc-gitlab-runner-certs-volumes# 卷volumes:# 卷類型- configMap:items:# configMap 中的鍵- key: gitlab.test.helm.xuxiaowei.cn.crt# 掛載的路徑path: gitlab.test.helm.xuxiaowei.cn.crtname: etc-gitlab-runner-certs# 配置 ConfigMap 名稱name: etc-gitlab-runner-certs-volumes

更新 helm 極狐GitLab 配置

helm upgrade -n gitlab-test --install my-gitlab gitlab/gitlab -f my-gitlab.yaml --timeout 600s --version 7.7.0

查看修改結果

kubectl -n gitlab-test get deployments.apps my-gitlab-gitlab-runner -o yaml

apiVersion: apps/v1
kind: Deployment
metadata:annotations:deployment.kubernetes.io/revision: "5"meta.helm.sh/release-name: my-gitlabmeta.helm.sh/release-namespace: gitlab-testcreationTimestamp: "2023-12-22T05:03:46Z"generation: 7labels:app: my-gitlab-gitlab-runnerapp.kubernetes.io/managed-by: Helmchart: gitlab-runner-0.59.2heritage: Helmrelease: my-gitlabname: my-gitlab-gitlab-runnernamespace: gitlab-testresourceVersion: "30086"uid: 8c46c44a-5b67-44ae-90d0-008daa3fa388
spec:progressDeadlineSeconds: 600replicas: 1revisionHistoryLimit: 10selector:matchLabels:app: my-gitlab-gitlab-runnerstrategy:rollingUpdate:maxSurge: 25%maxUnavailable: 25%type: RollingUpdatetemplate:metadata:annotations:checksum/configmap: f35865354f043583d0903b0a8350830a486eb0e289d18271cf3f533e7d89c5f7checksum/secrets: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855gitlab.com/prometheus_port: "9252"gitlab.com/prometheus_scrape: "true"creationTimestamp: nulllabels:app: my-gitlab-gitlab-runnerchart: gitlab-runner-0.59.2heritage: Helmrelease: my-gitlabspec:containers:- command:- /usr/bin/dumb-init- --- /bin/bash- /configmaps/entrypointenv:- name: CI_SERVER_URLvalue: https://gitlab.test.helm.xuxiaowei.cn- name: RUNNER_EXECUTORvalue: kubernetes- name: REGISTER_LOCKEDvalue: "false"- name: RUNNER_TAG_LIST- name: KUBERNETES_NAMESPACEvalue: gitlab-testimage: registry.gitlab.com/gitlab-org/gitlab-runner:alpine-v16.6.1imagePullPolicy: IfNotPresentlivenessProbe:exec:command:- /bin/bash- /configmaps/check-livefailureThreshold: 3initialDelaySeconds: 60periodSeconds: 10successThreshold: 1timeoutSeconds: 3name: my-gitlab-gitlab-runnerports:- containerPort: 9252name: metricsprotocol: TCPreadinessProbe:exec:command:- /usr/bin/pgrep- gitlab.*runnerfailureThreshold: 3initialDelaySeconds: 10periodSeconds: 10successThreshold: 1timeoutSeconds: 3resources: {}securityContext:allowPrivilegeEscalation: falsecapabilities:drop:- ALLprivileged: falsereadOnlyRootFilesystem: falserunAsNonRoot: trueterminationMessagePath: /dev/termination-logterminationMessagePolicy: FilevolumeMounts:- mountPath: /secretsname: projected-secrets- mountPath: /home/gitlab-runner/.gitlab-runnername: etc-gitlab-runner- mountPath: /configmapsname: configmaps- mountPath: /home/gitlab-runner/.gitlab-runner/certsname: etc-gitlab-runner-certs-volumesdnsPolicy: ClusterFirsthostAliases:- hostnames:- gitlab.test.helm.xuxiaowei.cnip: 172.25.25.32restartPolicy: AlwaysschedulerName: default-schedulersecurityContext:fsGroup: 65533runAsUser: 100serviceAccount: my-gitlab-gitlab-runnerserviceAccountName: my-gitlab-gitlab-runnerterminationGracePeriodSeconds: 3600volumes:- emptyDir:medium: Memoryname: runner-secrets- emptyDir:medium: Memoryname: etc-gitlab-runner- name: projected-secretsprojected:defaultMode: 420sources:- secret:name: my-gitlab-minio-secret- secret:items:- key: runner-registration-tokenpath: runner-registration-token- key: runner-tokenpath: runner-tokenname: my-gitlab-gitlab-runner-secret- configMap:defaultMode: 420name: my-gitlab-gitlab-runnername: configmaps- configMap:defaultMode: 420items:- key: gitlab.test.helm.xuxiaowei.cn.crtpath: gitlab.test.helm.xuxiaowei.cn.crtname: etc-gitlab-runner-certsname: etc-gitlab-runner-certs-volumes
status:availableReplicas: 1conditions:- lastTransitionTime: "2023-12-22T07:43:25Z"lastUpdateTime: "2023-12-22T07:43:25Z"message: Deployment has minimum availability.reason: MinimumReplicasAvailablestatus: "True"type: Available- lastTransitionTime: "2023-12-22T05:03:46Z"lastUpdateTime: "2023-12-22T07:43:25Z"message: ReplicaSet "my-gitlab-gitlab-runner-597d6d8f7c" has successfully progressed.reason: NewReplicaSetAvailablestatus: "True"type: ProgressingobservedGeneration: 7readyReplicas: 1replicas: 1updatedReplicas: 1

查看 gitlab runner 日志

[root@anolis-7-9 ~]# kubectl -n gitlab-test get pod | grep gitlab-runner
my-gitlab-gitlab-runner-597d6d8f7c-8v466             1/1     Running     0              5m52s
[root@anolis-7-9 ~]#

根據日志判斷，已經注冊成功了

[root@anolis-7-9 ~]# kubectl -n gitlab-test logs -f my-gitlab-gitlab-runner-597d6d8f7c-8v466
Registration attempt 1 of 30
Runtime platform                                    arch=amd64 os=linux pid=16 revision=f5da3c5a version=16.6.1
WARNING: Running in user-mode.                     
WARNING: The user-mode requires you to manually start builds processing:
WARNING: $ gitlab-runner run                       
WARNING: Use sudo for system-mode:                 
WARNING: $ sudo gitlab-runner...Created missing unique system ID                    system_id=r_3WsywNzJqRud
Merging configuration from template file "/configmaps/config.template.toml"
WARNING: Support for registration tokens and runner parameters in the 'register' command has been deprecated in GitLab Runner 15.6 and will be replaced with support for authentication tokens. For more information, see https://docs.gitlab.com/ee/ci/runners/new_creation_workflow
Registering runner... succeeded                     runner=BtGwLEwc
Runner registered successfully. Feel free to start it, but if it's running already the config should be automatically reloaded!Configuration (with the authentication token) was saved in "/home/gitlab-runner/.gitlab-runner/config.toml"
Runtime platform                                    arch=amd64 os=linux pid=7 revision=f5da3c5a version=16.6.1
Starting multi-runner from /home/gitlab-runner/.gitlab-runner/config.toml...  builds=0 max_builds=0
WARNING: Running in user-mode.                     
WARNING: Use sudo for system-mode:                 
WARNING: $ sudo gitlab-runner...There might be a problem with your config based on jsonschema annotations in common/config.go (experimental feature):
jsonschema: '/runners/0/kubernetes/pull_policy' does not validate with https://gitlab.com/gitlab-org/gitlab-runner/common/config#/$ref/properties/runners/items/$ref/properties/kubernetes/$ref/properties/pull_policy/$ref/type: expected array, but got nullConfiguration loaded                                builds=0 max_builds=10
listen_address not defined, metrics & debug endpoints disabled  builds=0 max_builds=10
[session_server].listen_address not defined, session endpoints disabled  builds=0 max_builds=10
Initializing executor providers                     builds=0 max_builds=10
^C
[root@anolis-7-9 ~]#

管理員訪問 https://gitlab.test.helm.xuxiaowei.cn/admin/runners 可以看到 pod my-gitlab-gitlab-runner-597d6d8f7c-8v466 已經注冊成了
1. 如果存在極狐 GitLab Runner Pod 被刪除（故障轉譯、手動刪除等等），這里可能會存在不可用的 Runner

問題2： tls: failed to verify certificate: x509: certificate is valid for ingress.local, not gitlab.test.helm.xuxiaowei.cn
1. 訪問的域名與使用的證書補匹配，可通過更換證書、域名來解決此問題（一般情況是更換證書）
2. 如果要更換證書：
  1. 證書屬于正規機構頒發的（如：上述各大云廠商的證書），請看下方方案2
  2. 如果是自己生成的證書（不受信任），請看下方方案2，并結合上方問題1來解決此問題

方案2：配置正規機構頒發的證書（如：上述各大云廠商的證書），一般無需在 gitlab 配置信任證書，即可正常使用

正常情況
1. 要求證書與域名對應
2. 將證書上傳至服務器，解壓，使用證書文件創建 k8s TLS Secret
3. 修改極狐GitLab 配置，使用上述步驟創建的 k8s TLS Secret
異常情況：正規機構頒發的證書，在極狐 GitLab Runner 中依然無法正常使用，但是在瀏覽器中可以正常使用
1. 原因是頒發機構的根證書不在極狐 GitLab Runner 的信任列表中，需要手動添加信任（例如：常見的 Windows 較低版本運行一些軟件時，也是無法執行，顯示證書無法識別，需要在 Windows 導入軟件簽名所使用的證書鏈）