Terraform Helm：微服務基礎設施即代碼

🚀 Terraform & Helm：微服務基礎設施即代碼

📚 目錄

🚀 Terraform & Helm：微服務基礎設施即代碼
- 1. 引言 🚀
- 2. 環境與依賴 🧰
- 3. 架構示意 🏗?
- 4. Terraform 定義云資源 🛠?
- - 4.1 Provider 與 Backend
  - 4.2 公共變量與 Tag
  - 4.3 Resource Group 模塊
  - 4.4 VNet 模塊
  - 4.5 Key Vault 模塊
  - 4.6 AKS 模塊
  - 4.7 RDS 模塊
  - 4.8 根模塊調用
- 5. Helm Chart 打包與校驗 📦
- - 5.1 Chart.yaml & Chart.lock
  - 5.2 values.schema.json
  - 5.3 Subchart 示例（gateway）
  - 5.4 Lint & Test
- 6. CI/CD 流水線 🔄
- - 6.1 Infra Workflow
  - 6.2 Deploy Workflow
- 7. 可觀測性與告警 🔔
- 8. 附錄 📂

1. 引言 🚀

? TL;DR

🛠? 精準聲明：Terraform 管理 Resource Group、VNet（多子網）、AKS（跨可用區 + Azure Monitor）、Key Vault、PostgreSQL 等生產資源，統一 Tag 并鎖定 Provider 版本
📦 Umbrella Chart：Helm Umbrella Chart 支持多微服務（Gateway、Identity…），含 Probes、PodDisruptionBudget、NetworkPolicy、HPA、Secret 引用、values.schema.json 與 Chart.lock
🔄 端到端 CI/CD：GitHub Actions 流水線集成 OIDC 登錄、Terraform fmt/validate/lint、Checkov、Infracost、Azure 登錄、ACR 鑒權、Helm lint/test/package/push、自動回滾，并發控制與環境審批
🌟 企業級要素：高性能（Azure CNI + HPA + PDB）、高可用（多 AZ + Monitoring）、安全可復現（Key Vault + Terraform Backend + Sensitive + OIDC + Cost Scan）

📚 背景與動機
在多微服務架構下，“在我機器上沒問題”往往難以復現🧩。通過 Terraform + Helm Chart + GitOps/CI-CD，可實現基礎設施與應用部署一體化自動化、一致化、可審計、可回滾，大幅提升交付速度與可靠性💪。

2. 環境與依賴 🧰

terraform version      # >=1.4
az version             # 最新 Azure CLI
kubectl version --client
helm version           # >=3.8+

倉庫結構

.
├─ infra/
│   ├─ terraform/
│   │    ├─ backend.tf
│   │    ├─ required_providers.tf
│   │    ├─ variables.tf
│   │    ├─ main.tf
│   │    ├─ outputs.tf
│   │    └─ modules/
│   │         ├─ resource_group/
│   │         ├─ vnet/
│   │         ├─ keyvault/
│   │         ├─ aks/
│   │         └─ rds/
│   └─ helm-charts/
│        └─ abp-vnext/
│             ├─ Chart.yaml
│             ├─ Chart.lock
│             ├─ values.schema.json
│             ├─ values.yaml
│             ├─ values-dev.yaml
│             ├─ values-prod.yaml
│             ├─ charts/           # Gateway、Identity 等 Subcharts
│             └─ templates/        # Umbrella 公共資源（Ingress 可選）
└─ src/└─ MyAbpSolution.sln

3. 架構示意 🏗?

4. Terraform 定義云資源 🛠?

4.1 Provider 與 Backend

# required_providers.tf
terraform {required_version = ">= 1.4"required_providers {azurerm = {source  = "hashicorp/azurerm"version = "~> 3.0"}}backend "azurerm" {resource_group_name  = "tfstate-rg"storage_account_name = "tfstateacct"container_name       = "tfstate"key                  = "${terraform.workspace}.tfstate"}
}

4.2 公共變量與 Tag

# variables.tf
variable "location"    { type = string }
variable "environment" { type = string }
variable "owner"       { type = string }
variable "cost_center" { type = string }
variable "db_admin"    { type = string }
variable "db_password" { type = string }locals {common_tags = {environment = var.environmentowner       = var.ownercost_center = var.cost_center}
}

4.3 Resource Group 模塊

# modules/resource_group/main.tf
resource "azurerm_resource_group" "this" {name     = var.namelocation = var.locationtags     = var.tags
}
output "rg_name" { value = azurerm_resource_group.this.name }

# modules/resource_group/variables.tf
variable "name"     { type = string }
variable "location" { type = string }
variable "tags"     { type = map(string) }

4.4 VNet 模塊

# modules/vnet/main.tf
resource "azurerm_virtual_network" "this" {name                = var.nameaddress_space       = var.address_spacelocation            = var.locationresource_group_name = var.rg_nametags                = var.tags
}resource "azurerm_subnet" "this" {for_each            = var.subnetsname                = each.keyresource_group_name = var.rg_namevirtual_network_name= azurerm_virtual_network.this.nameaddress_prefixes    = [each.value]
}output "subnet_ids_map" {value = { for s in azurerm_subnet.this : s.name => s.id }
}

# modules/vnet/variables.tf
variable "name"          { type = string }
variable "location"      { type = string }
variable "rg_name"       { type = string }
variable "address_space" { type = list(string) }
variable "subnets"       { type = map(string) }
variable "tags"          { type = map(string) }

4.5 Key Vault 模塊

# modules/keyvault/main.tf
data "azurerm_client_config" "current" {}resource "azurerm_key_vault" "this" {name                        = var.namelocation                    = var.locationresource_group_name         = var.rg_nametenant_id                   = data.azurerm_client_config.current.tenant_idsku_name                    = "standard"purge_protection_enabled    = truesoft_delete_enabled         = truetags                        = var.tags
}resource "azurerm_key_vault_secret" "db_password" {name         = "db-password"value        = var.admin_passwordkey_vault_id = azurerm_key_vault.this.id
}

# modules/keyvault/variables.tf
variable "name"           { type = string }
variable "location"       { type = string }
variable "rg_name"        { type = string }
variable "admin_password" { type = string }
variable "tags"           { type = map(string) }

4.6 AKS 模塊

# modules/aks/main.tf
resource "azurerm_kubernetes_cluster" "this" {name                = var.namelocation            = var.locationresource_group_name = var.rg_namedns_prefix          = var.dns_prefixtags                = var.tagsdefault_node_pool {name                = "agentpool"vm_size             = var.node_sizeavailability_zones  = var.availability_zonesenable_auto_scaling = var.enable_auto_scalermin_count           = var.min_countmax_count           = var.max_countos_disk_size_gb     = 50}network_profile {network_plugin    = "azure"network_policy    = "calico"load_balancer_sku = "standard"subnet_id         = var.aks_subnet_id}identity { type = "SystemAssigned" }addon_profile { oms_agent { enabled = true } }  # Azure Monitor
}output "kube_config" {value     = azurerm_kubernetes_cluster.this.kube_admin_config_rawsensitive = true
}

# modules/aks/variables.tf
variable "name"               { type = string }
variable "location"           { type = string }
variable "rg_name"            { type = string }
variable "dns_prefix"         { type = string }
variable "aks_subnet_id"      { type = string }
variable "node_size"          { type = string }
variable "availability_zones"{ type = list(string), default = ["1","2","3"] }
variable "enable_auto_scaler" { type = bool, default = true }
variable "min_count"          { type = number, default = 2 }
variable "max_count"          { type = number, default = 5 }
variable "tags"               { type = map(string) }

4.7 RDS 模塊

# modules/rds/main.tf
data "azurerm_key_vault_secret" "db_pwd" {name         = "db-password"key_vault_id = var.keyvault_id
}resource "azurerm_postgresql_flexible_server" "this" {name                         = var.namelocation                     = var.locationresource_group_name          = var.rg_nameversion                      = var.pg_versionsku_name                     = var.sku_namestorage_mb                   = var.storage_mbdelegated_subnet_id          = var.subnet_idadministrator_login          = var.admin_useradministrator_login_password = data.azurerm_key_vault_secret.db_pwd.valuetags                         = var.tags
}output "connection_string" {value     = format("Host=%s;Port=5432;Username=%s;Password=%s;Database=%s",azurerm_postgresql_flexible_server.this.fqdn,var.admin_user,data.azurerm_key_vault_secret.db_pwd.value,var.db_name)sensitive = true
}

# modules/rds/variables.tf
variable "name"        { type = string }
variable "location"    { type = string }
variable "rg_name"     { type = string }
variable "pg_version"  { type = string }
variable "sku_name"    { type = string }
variable "storage_mb"  { type = number }
variable "admin_user"  { type = string }
variable "subnet_id"   { type = string }
variable "db_name"     { type = string }
variable "keyvault_id" { type = string }
variable "tags"        { type = map(string) }

4.8 根模塊調用

# main.tf
provider "azurerm" { features {} }module "rg" {source   = "./modules/resource_group"name     = "${var.environment}-rg"location = var.locationtags     = local.common_tags
}module "vnet" {source        = "./modules/vnet"name          = "${var.environment}-vnet"location      = var.locationrg_name       = module.rg.rg_nameaddress_space = ["10.0.0.0/16"]subnets       = { aks = "10.0.1.0/24", db = "10.0.2.0/24" }tags          = local.common_tags
}module "keyvault" {source         = "./modules/keyvault"name           = "${var.environment}-kv"location       = var.locationrg_name        = module.rg.rg_nameadmin_password = var.db_passwordtags           = local.common_tags
}module "aks" {source            = "./modules/aks"name              = "${var.environment}-aks"location          = var.locationrg_name           = module.rg.rg_namedns_prefix        = var.environmentaks_subnet_id     = module.vnet.subnet_ids_map["aks"]node_size         = "Standard_DS2_v2"availability_zones= ["1","2","3"]enable_auto_scaler= truemin_count         = 2max_count         = 5tags              = local.common_tags
}module "rds" {source      = "./modules/rds"name        = "${var.environment}-pg"location    = var.locationrg_name     = module.rg.rg_namepg_version  = "13"sku_name    = "GP_Gen5_2"storage_mb  = 5120admin_user  = var.db_adminsubnet_id   = module.vnet.subnet_ids_map["db"]keyvault_id = module.keyvault.azurerm_key_vault.this.iddb_name     = "abpdb"tags        = local.common_tags
}

# outputs.tf
output "kubeconfig"     { value = module.aks.kube_config      sensitive = true }
output "db_conn_string" { value = module.rds.connection_string sensitive = true }

5. Helm Chart 打包與校驗 📦

5.1 Chart.yaml & Chart.lock

# Chart.yaml
apiVersion: v2
name: abp-vnext
version: 0.4.0
appVersion: "1.0.0"
description: "ABP VNext 多服務 Kubernetes Umbrella Chart"
dependencies:- name: gatewayversion: "0.2.0"repository: file://charts/gateway- name: identityversion: "0.2.0"repository: file://charts/identity

helm dependency update infra/terraform/helm-charts/abp-vnext
helm dependency build  infra/terraform/helm-charts/abp-vnext

5.2 values.schema.json

{"$schema": "https://json-schema.org/draft/2020-12/schema","type": "object","properties": {"replicaCount": { "type": "integer" },"image": {"type": "object","properties": {"repository": { "type": "string" },"tag": { "type": "string" }},"required": ["repository","tag"]},"service": {"type": "object","properties": {"type": { "type": "string" },"port": { "type": "integer" }},"required": ["type","port"]}},"required": ["replicaCount","image","service"]
}

5.3 Subchart 示例（gateway）

# charts/gateway/values.yaml
replicaCount: 2
image:repository: myacr.azurecr.io/abp-gatewaytag: "1.0.0"
service:type: ClusterIPport: 80
resources:limits:cpu: "500m"memory: "512Mi"requests:cpu: "250m"memory: "256Mi"

# charts/gateway/templates/deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:name: {{ include "gateway.fullname" . }}
spec:replicas: {{ .Values.replicaCount }}selector:matchLabels:app: {{ include "gateway.name" . }}template:metadata:labels:app: {{ include "gateway.name" . }}spec:containers:- name: gatewayimage: "{{ .Values.image.repository }}:{{ .Values.image.tag }}"ports:- containerPort: 80readinessProbe:httpGet:path: /health/readyport: 80initialDelaySeconds: 20periodSeconds: 10livenessProbe:httpGet:path: /health/liveport: 80initialDelaySeconds: 30periodSeconds: 15resources:{{ toYaml .Values.resources | indent 12 }}

其余 Service、Ingress、PDB、NetworkPolicy、HPA 與前文一致。

5.4 Lint & Test

helm lint infra/terraform/helm-charts/abp-vnext --strict
helm template abp-vnext infra/terraform/helm-charts/abp-vnext
helm package  infra/terraform/helm-charts/abp-vnext -d charts-packages

6. CI/CD 流水線 🔄

6.1 Infra Workflow

# .github/workflows/infra.yml
name: Infra – Terraformon:push:paths: ["infra/terraform/**"]concurrency:group: infra-${{ github.ref }}cancel-in-progress: truepermissions:id-token: writecontents: readjobs:terraform:runs-on: ubuntu-latestoutputs:kubeconfig: ${{ steps.apply.outputs.kubeconfig }}db_conn_string: ${{ steps.apply.outputs.db_conn_string }}steps:- uses: actions/checkout@v3- name: Setup Terraformuses: hashicorp/setup-terraform@v2- name: Terraform Fmt Checkrun: terraform fmt -checkworking-directory: infra/terraform- name: Terraform Validaterun: terraform validateworking-directory: infra/terraform- name: Terraform Initrun: terraform init -input=falseworking-directory: infra/terraform- name: Checkov Scanuses: bridgecrewio/checkov-action@masterwith:directory: infra/terraform- name: Infracost Estimateuses: infracost/actions@v2with:path: infra/terraformenv:INFRACOST_TOKEN: ${{ secrets.INFRACOST_TOKEN }}- name: Terraform Planid: planrun: terraform plan -out=tfplanworking-directory: infra/terraform- name: Terraform Applyid: applyrun: |terraform workspace select ${{ github.ref_name }} || terraform workspace new ${{ github.ref_name }}terraform apply -auto-approve tfplanecho "kubeconfig<<EOF" >> $GITHUB_OUTPUTterraform output -raw kubeconfig >> $GITHUB_OUTPUTecho "EOF" >> $GITHUB_OUTPUTecho "db_conn_string<<EOF" >> $GITHUB_OUTPUTterraform output -raw db_conn_string >> $GITHUB_OUTPUTecho "EOF" >> $GITHUB_OUTPUTworking-directory: infra/terraformenv:ARM_USE_MSI:          trueARM_SUBSCRIPTION_ID:  ${{ secrets.AZURE_SUBSCRIPTION_ID }}ARM_TENANT_ID:        ${{ secrets.AZURE_TENANT_ID }}

6.2 Deploy Workflow

# .github/workflows/deploy.yml
name: Deploy – Helmon:push:paths:- "src/**"- "infra/terraform/helm-charts/**"concurrency:group: deploy-${{ github.ref }}cancel-in-progress: truepermissions:id-token: writecontents: readenvironment:name: productionurl: https://abp.example.comreviewers:- alice- bobjobs:deploy:runs-on: ubuntu-latestneeds: terraformsteps:- uses: actions/checkout@v3- name: Azure Login via OIDCuses: azure/login@v1with:client-id:       ${{ secrets.AZURE_CLIENT_ID }}tenant-id:       ${{ secrets.AZURE_TENANT_ID }}subscription-id: ${{ secrets.AZURE_SUBSCRIPTION_ID }}- name: AKS Set Contextuses: azure/aks-set-context@v1with:resource-group: production-rgcluster-name:   production-aks- name: Docker Login to ACRuses: docker/login-action@v2with:registry: myacr.azurecr.iousername: ${{ secrets.ACR_USERNAME }}password: ${{ secrets.ACR_PASSWORD }}- name: Build & Push Imagerun: |docker build -t myacr.azurecr.io/abp-vnext:${{ github.sha }} src/docker push myacr.azurecr.io/abp-vnext:${{ github.sha }}- name: Set DEPLOY_ENVrun: |if [ "${GITHUB_REF}" == "refs/heads/main" ]; thenecho "DEPLOY_ENV=prod" >> $GITHUB_ENVelseecho "DEPLOY_ENV=dev" >> $GITHUB_ENVfi- name: Helm Lint & Testrun: |helm lint infra/terraform/helm-charts/abp-vnext --stricthelm test --cleanup gateway- name: Helm Package & Pushrun: |helm package infra/terraform/helm-charts/abp-vnext -d charts-packageshelm push charts-packages/abp-vnext-*.tgz oci://myhelmrepo- name: Helm Upgrade with Rollbackrun: |set +ehelm repo add myrepo oci://myhelmrepohelm repo updatehelm upgrade --install abp-vnext myrepo/abp-vnext \--version 0.4.0 \-f infra/terraform/helm-charts/abp-vnext/values-${DEPLOY_ENV}.yaml \--set image.tag=${GITHUB_SHA} \--set-string env.DB_CONN="${{ needs.terraform.outputs.db_conn_string }}" \--wait --timeout 5mif [ $? -ne 0 ]; thenhelm rollback abp-vnext 1exit 1fiset -e- name: Verify Rolloutrun: kubectl rollout status deployment/gateway- name: Notify Slack on Successif: success()uses: 8398a7/action-slack@v3with:payload: '{"text":"? 部署成功：ABP VNext 微服務已更新到 '"${{ github.sha }}"'"}'channel: production-alertsenv:SLACK_WEBHOOK_URL: ${{ secrets.SLACK_WEBHOOK_URL }}- name: Notify Slack on Failureif: failure()uses: 8398a7/action-slack@v3with:payload: '{"text":"? 部署失敗：請檢查流水線日志！"}'channel: production-alertsenv:SLACK_WEBHOOK_URL: ${{ secrets.SLACK_WEBHOOK_URL }}