什么是 有狀態服務和 無狀態服務
有狀態服務(Stateful Service):
有狀態服務是指在處理請求期間維護和跟蹤用戶狀態或會話信息的服務。這意味著服務在多個請求之間保持狀態,并且需要在請求之間共享和使用這些狀態信息。通常,有狀態服務會將用戶數據存儲在內存、數據庫或其他持久化存儲中,并使用該狀態來處理后續請求。有狀態服務通常需要進行會話管理和狀態同步,以確保正確處理和維護用戶狀態。
無狀態服務(Stateless Service):
無狀態服務是指在處理請求期間不維護任何用戶狀態或會話信息的服務。每個請求被視為獨立的、無關的操作,服務不會存儲或依賴之前的請求狀態。無狀態服務僅根據當前請求的輸入進行處理,并生成相應的輸出。無狀態服務更加簡單和可伸縮,因為它們不需要跟蹤和管理用戶狀態。每個請求都是獨立的,可以在集群中的任何實例之間進行負載均衡。
總結:
有狀態服務維護和使用用戶狀態或會話信息,需要進行狀態同步和會話管理。
無狀態服務不維護任何用戶狀態或會話信息,每個請求是獨立的。
再簡單地講, 有狀態服務需要考慮用戶數據如何在各個service instance 中同步的問題。
而無狀態服務是不需要考慮的, 所以無狀態服務能基于QPS 無限制地進行增加實例, 但是有狀態并不能這么做
K8S 的deployment 是不是只能用于無狀態服務
Deployment 在 Kubernetes 中用于管理無狀態應用的更新和擴縮容,但它也可以用于管理有狀態應用。Deployment 控制器是 Kubernetes 中的一種資源對象,它提供了應用部署的聲明性描述,并負責確保所需的 Pod 副本數目在集群中運行。
Deployment 控制器的主要功能是實現滾動更新(Rolling Update)和回滾(Rollback)。無狀態應用通常使用 Deployment 來進行部署和更新,因為它們的實例可以平滑地替換,而不會對應用的狀態造成影響。
然而,有狀態應用也可以使用 Deployment 進行部署,尤其是在需要水平擴展和滾動更新有狀態應用時。通過配置適當的存儲卷(Volume)和持久性聲明(PersistentVolumeClaim),可以確保有狀態應用在 Pod 替換過程中不會丟失數據。
此外,對于有狀態應用,Kubernetes 還提供了 StatefulSet 控制器,它專門用于部署和管理有狀態應用。StatefulSet 可以為每個 Pod 分配一個唯一的標識符和穩定的網絡標識,確保有狀態應用的穩定性和持久性。StatefulSet 適用于需要有序部署、有狀態存儲和有狀態網絡標識的應用場景。
因此,Deployment 對于無狀態應用是常用的部署方式,但也可以用于管理有狀態應用。對于有狀態應用,可以使用 StatefulSet 來獲得更多的特性和保證。
StatefulSet 更適合用于有狀態service的部署
原因如下:
-
穩定的網絡標識符:StatefulSet 為每個 Pod 分配一個唯一的標識符,通常是以序號的形式命名,例如 myapp-0, myapp-1,以此類推。這些標識符在 Pod 的重新創建過程中保持不變,使得有狀態應用能夠保持穩定的網絡標識。這對于一些需要固定標識符的應用非常重要,例如數據庫集群中的主從關系或者分布式系統中的節點標識。
-
有序的部署和擴縮容:StatefulSet 支持有序的部署和擴縮容。在擴展 StatefulSet 時,Kubernetes 會按照指定的順序創建新的 Pod,確保先創建的 Pod 首先可用。這對于一些有狀態應用非常重要,例如數據庫集群中的主從復制關系,需要確保先啟動主節點再啟動從節點。
-
持久性存儲:StatefulSet 可以與持久性存儲卷(Persistent Volume)和持久性卷聲明(Persistent Volume Claim)結合使用,確保有狀態應用在 Pod 的重新創建過程中保留數據。每個 Pod 可以綁定到一個獨立的持久性存儲卷,以確保數據的持久性和可靠性。
-
有狀態應用的管理:StatefulSet 提供了一些管理有狀態應用的功能。例如,可以使用 rollingUpdate 策略來控制有狀態應用的滾動更新過程,確保更新的穩定性。此外,StatefulSet 還支持有狀態應用的有序刪除,以及與有狀態應用相關的服務發現和 DNS 解析。
一個StatefulSet 的部署例子 - Creation
先編寫1個yaml file:
stateful-nginx-without-pvc.yaml:
---
apiVersion: v1 # api version
kind: Service # type of this resource e.g. Pod/Deployment ..
metadata:name: nginx-stateful-service # name of the servicelabels:app: nginx-stateful-service
spec:ports: - port: 80 # port of the service, used to access the servicename: web-portclusterIP: None # the service is not exposed outside the clusterselector: # label of the Pod that the Service is selectingapp: nginx-stateful # only service selector could skip the matchLabels:
---apiVersion: apps/v1
kind: StatefulSet # it's for a stateful application, it's a controller
metadata:name: nginx-statefulset # name of the statefulsetlabels:app: nginx-stateful
spec: # detail descriptionserviceName: "nginx-stateful-service" # name of the service that used to manange the dns, # must be the same as the service name defined abovereplicas: 3 # desired replica countselector: # label of the Pod that the StatefulSet is managingmatchLabels:app: nginx-statefultemplate: # Pod templatemetadata:labels:app: nginx-statefulspec:containers:- name: nginx-containerimage: nginx:1.25.4 # image of the containerports: # the ports of the container and they will be exposed- containerPort: 80 # the port used by the container servicename: web-port
上面的yaml 文件創建了兩個資源
1個是statefulset 部署對象
1個是service
但是并沒有pvc data volume 對象, 嚴格上將上面部署的service還是無狀態的, 但是本部分之focus on statefulset 的creation, 先忽略pvc.
置于為何還需要1個service
關鍵就是下面service name 的標簽
serviceName: "nginx-stateful-service" # name of the service that used to manange the dns, # must be the same as the service name defined above
serviceName 字段的作用是為 StatefulSet 中的每個 Pod 提供一個穩定的網絡標識符。當 StatefulSet 創建 Pod 時,每個 Pod 都會以其索引號作為后綴,形成一個唯一的 DNS 名稱。這個 DNS 名稱由以下組成:pod-name.service-name.namespace.svc.cluster.local。
通過指定 serviceName 字段,StatefulSet 與服務進行關聯,使得每個 Pod 都能夠使用服務的名稱作為其 DNS 標識符。這樣,每個 Pod 可以通過服務名稱進行網絡通信,而不需要知道其他 Pod 的具體名稱或 IP 地址。
當執行這個配置文件后, 可以見到 pods 是順序創建的, 創建好 pod0 再是 pod1 才是 pod2, 這就是1個和deployment pods創建很大的區別!
[gateman@manjaro-x13 statefulsets]$ kubectl apply -f stateful-nginx-without-pvc.yaml
service/nginx-stateful-service created
statefulset.apps/nginx-statefulset created
[gateman@manjaro-x13 statefulsets]$ bash show.sh
+ kubectl get pvc
No resources found in default namespace.
+ kubectl get svc
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
bq-api-service-1 NodePort 10.103.40.130 <none> 32111:30604/TCP 15h
kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 76d
nginx-stateful-service ClusterIP None <none> 80/TCP 6s
+ kubectl get sts
NAME READY AGE
nginx-statefulset 1/3 6s
+ kubectl get pods
NAME READY STATUS RESTARTS AGE
bq-api-service-deployment-978b76fcf-4qcsp 1/1 Running 0 5h20m
bq-api-service-deployment-978b76fcf-7x54w 1/1 Running 0 5h20m
bq-api-service-deployment-978b76fcf-xfxcw 1/1 Running 0 5h20m
nginx-statefulset-0 1/1 Running 0 7s
nginx-statefulset-1 0/1 ContainerCreating 0 5s
[gateman@manjaro-x13 statefulsets]$
[gateman@manjaro-x13 statefulsets]$ bash show.sh
+ kubectl get pvc
No resources found in default namespace.
+ kubectl get svc
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
bq-api-service-1 NodePort 10.103.40.130 <none> 32111:30604/TCP 15h
kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 76d
nginx-stateful-service ClusterIP None <none> 80/TCP 15s
+ kubectl get sts
NAME READY AGE
nginx-statefulset 2/3 15s
+ kubectl get pods
NAME READY STATUS RESTARTS AGE
bq-api-service-deployment-978b76fcf-4qcsp 1/1 Running 0 5h20m
bq-api-service-deployment-978b76fcf-7x54w 1/1 Running 0 5h20m
bq-api-service-deployment-978b76fcf-xfxcw 1/1 Running 0 5h20m
nginx-statefulset-0 1/1 Running 0 16s
nginx-statefulset-1 1/1 Running 0 14s
nginx-statefulset-2 0/1 ContainerCreating 0 7s
[gateman@manjaro-x13 statefulsets]$ bash show.sh
+ kubectl get pvc
No resources found in default namespace.
+ kubectl get svc
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
bq-api-service-1 NodePort 10.103.40.130 <none> 32111:30604/TCP 15h
kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 76d
nginx-stateful-service ClusterIP None <none> 80/TCP 52s
+ kubectl get sts
NAME READY AGE
nginx-statefulset 3/3 52s
+ kubectl get pods
NAME READY STATUS RESTARTS AGE
bq-api-service-deployment-978b76fcf-4qcsp 1/1 Running 0 5h20m
bq-api-service-deployment-978b76fcf-7x54w 1/1 Running 0 5h20m
bq-api-service-deployment-978b76fcf-xfxcw 1/1 Running 0 5h20m
nginx-statefulset-0 1/1 Running 0 53s
nginx-statefulset-1 1/1 Running 0 51s
nginx-statefulset-2 1/1 Running 0 44s
重點, 跟上面的deployments的pod是不同, deployment pods 是無序的, 后面跟的是無意義的字符串, 而statefulset 的pods 是嚴格按照順序創建! POD名字后序是index 數字
測試service
當service 創建之后
默認在 nodes里是無法直接訪問的, 例如
root@k8s-master:~# ping nginx-statefulset-0.nginx-stateful-service
ping: nginx-statefulset-0.nginx-stateful-service: Name or service not known
root@k8s-master:~# ping nginx-statefulset-0.nginx-stateful-service.default.svc.cluster.local
ping: nginx-statefulset-0.nginx-stateful-service.default.svc.cluster.local: Name or service not known
上面提到了, 這個例子中的service 的作用是為pods 提供1個穩定的DNS name
但是無論是
簡寫: $podName.$serviceName
nginx-statefulset-0.nginx-stateful-service
還是全寫:$podName.$serviceName.$nameSpace.svc.cluster.local
nginx-statefulset-0.nginx-stateful-service.default.svc.cluster.local
在k8s-master 中是無法訪問的
原因就是這些DNS 只能在 k8s 的pod內部訪問!
為了測試, 我們臨時創建1個pod with busybox
kubectl run -it --image busybox:1.28 dns-test --restart=Never /bin/sh
注意, busybox 默認是沒安裝bash 和 curl的, 但是有ping 和 nslookup
現在可以ping 通了
/ # ping nginx-statefulset-0.nginx-stateful-service
PING nginx-statefulset-0.nginx-stateful-service (10.244.2.109): 56 data bytes
64 bytes from 10.244.2.109: seq=0 ttl=64 time=14.367 ms
64 bytes from 10.244.2.109: seq=1 ttl=64 time=0.093 ms
nslookup 結果
可見 k8s 容器內 , DNS server 是10.96.0.10 kube-dns.kube-system.svc.cluster.local
它會把nginx-statefulset-0.nginx-stateful-service
解釋為 10.244.2.109 nginx-statefulset-0.nginx-stateful-service.default.svc.cluster.local
而這個10.244.2.109 就是該pod的ip, 并且給了1個DNS 域名 nginx-statefulset-0.nginx-stateful-service.default.svc.cluster.local
/ # nslookup nginx-statefulset-0.nginx-stateful-service
Server: 10.96.0.10
Address 1: 10.96.0.10 kube-dns.kube-system.svc.cluster.localName: nginx-statefulset-0.nginx-stateful-service
Address 1: 10.244.2.109 nginx-statefulset-0.nginx-stateful-service.default.svc.cluster.local
/ # nslookup nginx-statefulset-1.nginx-stateful-service
Server: 10.96.0.10
Address 1: 10.96.0.10 kube-dns.kube-system.svc.cluster.localName: nginx-statefulset-1.nginx-stateful-service
Address 1: 10.244.3.62 nginx-statefulset-1.nginx-stateful-service.default.svc.cluster.local
/ # nslookup nginx-statefulset-2.nginx-stateful-service
Server: 10.96.0.10
Address 1: 10.96.0.10 kube-dns.kube-system.svc.cluster.localName: nginx-statefulset-2.nginx-stateful-service
Address 1: 10.244.1.53 nginx-statefulset-2.nginx-stateful-service.default.svc.cluster.local
在容器內用該域名是可以訪問ngnix服務的
/ # wget http://nginx-statefulset-0.nginx-stateful-service.default.svc.cluster.local
Connecting to nginx-statefulset-0.nginx-stateful-service.default.svc.cluster.local (10.244.2.109:80)
index.html 100% |******************************************************************************************************************************************************************| 615 0:00:00 ETA
/ # cat index.html
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
html { color-scheme: light dark; }
body { width: 35em; margin: 0 auto;
font-family: Tahoma, Verdana, Arial, sans-serif; }
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p><p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p><p><em>Thank you for using nginx.</em></p>
</body>
</html>
StatefulSet 的scale - 擴容和縮容
命令有兩種
kubectl scale statefulset \$statefulsetName --replicas=5
# another method
kubectl patch statefulset statefulsetName -p '{"spec":{"replicas":5}}'
兩種命令都可以用來擴容和縮容
例如:
用第一命令去把 replicas 從3 改成10
[gateman@manjaro-x13 statefulsets]$ kubectl scale statefulset nginx-statefulset --replicas=10
statefulset.apps/nginx-statefulset scaled
[gateman@manjaro-x13 statefulsets]$
[gateman@manjaro-x13 statefulsets]$ kubectl describe sts nginx-statefulset
Name: nginx-statefulset
Namespace: default
CreationTimestamp: Sat, 22 Jun 2024 19:48:11 +0800
Selector: app=nginx-stateful
Labels: app=nginx-stateful
Annotations: <none>
Replicas: 10 desired | 9 total
Update Strategy: RollingUpdatePartition: 0
Pods Status: 9 Running / 1 Waiting / 0 Succeeded / 0 Failed
Pod Template:Labels: app=nginx-statefulContainers:nginx-container:Image: nginx:1.25.4Port: 80/TCPHost Port: 0/TCPEnvironment: <none>Mounts: <none>Volumes: <none>Node-Selectors: <none>Tolerations: <none>
Volume Claims: <none>
Events:Type Reason Age From Message---- ------ ---- ---- -------Normal SuccessfulCreate 36s (x2 over 98s) statefulset-controller create Pod nginx-statefulset-3 in StatefulSet nginx-statefulset successfulNormal SuccessfulCreate 35s (x2 over 97s) statefulset-controller create Pod nginx-statefulset-4 in StatefulSet nginx-statefulset successfulNormal SuccessfulCreate 34s statefulset-controller create Pod nginx-statefulset-5 in StatefulSet nginx-statefulset successfulNormal SuccessfulCreate 32s statefulset-controller create Pod nginx-statefulset-6 in StatefulSet nginx-statefulset successfulNormal SuccessfulCreate 31s statefulset-controller create Pod nginx-statefulset-7 in StatefulSet nginx-statefulset successfulNormal SuccessfulCreate 29s statefulset-controller create Pod nginx-statefulset-8 in StatefulSet nginx-statefulset successfulNormal SuccessfulCreate 28s statefulset-controller create Pod nginx-statefulset-9 in StatefulSet nginx-statefulset successful
通過describe 命令 可以見 當前運行的副本的確變成10 (9 running 還有1個正在創建)
而且pods 的創建是按順序執行的
StatefulSet 的rollingUpdate 滾動更新
跟deployment 一樣, 一樣可以用set image 來執行
[gateman@manjaro-x13 bq-api-service]$ kubectl set image statefulset/nginx-statefulset nginx-container=nginx:1.26.1
statefulset.apps/nginx-statefulset image updated
從describe 信息來看, rolling update 也是按照pod的順序來執行的
Events:Type Reason Age From Message---- ------ ---- ---- -------Normal SuccessfulCreate 13m (x2 over 23m) statefulset-controller create Pod nginx-statefulset-9 in StatefulSet nginx-statefulset successfulNormal SuccessfulCreate 13m (x2 over 23m) statefulset-controller create Pod nginx-statefulset-8 in StatefulSet nginx-statefulset successfulNormal SuccessfulDelete 13m statefulset-controller delete Pod nginx-statefulset-7 in StatefulSet nginx-statefulset successfulNormal SuccessfulCreate 13m (x2 over 23m) statefulset-controller create Pod nginx-statefulset-7 in StatefulSet nginx-statefulset successfulNormal SuccessfulDelete 12m statefulset-controller delete Pod nginx-statefulset-6 in StatefulSet nginx-statefulset successfulNormal SuccessfulCreate 12m (x2 over 23m) statefulset-controller create Pod nginx-statefulset-6 in StatefulSet nginx-statefulset successfulNormal SuccessfulCreate 12m (x2 over 24m) statefulset-controller create Pod nginx-statefulset-5 in StatefulSet nginx-statefulset successfulNormal SuccessfulDelete 12m statefulset-controller delete Pod nginx-statefulset-5 in StatefulSet nginx-statefulset successfulNormal SuccessfulDelete 12m (x2 over 24m) statefulset-controller delete Pod nginx-statefulset-4 in StatefulSet nginx-statefulset successfulNormal SuccessfulCreate 12m (x3 over 25m) statefulset-controller create Pod nginx-statefulset-4 in StatefulSet nginx-statefulset successfulNormal SuccessfulDelete 12m (x2 over 24m) statefulset-controller delete Pod nginx-statefulset-3 in StatefulSet nginx-statefulset successfulNormal SuccessfulCreate 12m (x3 over 25m) statefulset-controller create Pod nginx-statefulset-3 in StatefulSet nginx-statefulset successfulNormal SuccessfulDelete 12m statefulset-controller delete Pod nginx-statefulset-2 in StatefulSet nginx-statefulset successfulNormal SuccessfulCreate 12m (x2 over 3h) statefulset-controller create Pod nginx-statefulset-2 in StatefulSet nginx-statefulset successful
一樣可以用kubectl annotate 命令去提供change cause
[gateman@manjaro-x13 bq-api-service]$ kubectl annotate statefulset/nginx-statefulset kubernetes.io/change-cause="downgraded to 1.26.1"
statefulset.apps/nginx-statefulset annotated
但是很可惜, 跟deployment 不一樣, 不能在revision list 里顯示 change cause的annotation
[gateman@manjaro-x13 bq-api-service]$ kubectl rollout history statefulset/nginx-statefulset
statefulset.apps/nginx-statefulset
REVISION CHANGE-CAUSE
1 <none>
2 <none>
而且跟deployment 不同
statefulset的 revision 沒有 replicasets 的對應
[gateman@manjaro-x13 statefulsets]$ kubectl get rs -o wide
NAME DESIRED CURRENT READY AGE CONTAINERS IMAGES SELECTOR
bq-api-service-deployment-6f6ffc7866 0 0 0 18h bq-api-service-container europe-west2-docker.pkg.dev/jason-hsbc/my-docker-repo/bq-api-service:1.1.7 app=bq-api-service,pod-template-hash=6f6ffc7866
bq-api-service-deployment-978b76fcf 3 3 3 8h bq-api-service-container europe-west2-docker.pkg.dev/jason-hsbc/my-docker-repo/bq-api-service:1.1.8 app=bq-api-service,pod-template-hash=978b76fcf
bq-api-service-deployment-c4979b697 0 0 0 18h bq-api-service-container europe-west2-docker.pkg.dev/jason-hsbc/my-docker-repo/bq-api-service:1.1.6 app=bq-api-service,pod-template-hash=c4979b697
只有deployment的record