openelb的介紹
具體根據官方文檔進行安裝官方文檔,這里作為測試環境的安裝使用.
OpenELB 是一個開源的云原生負載均衡器實現,可以在基于裸金屬服務器、邊緣以及虛擬化的 Kubernetes 環境中使用 LoadBalancer 類型的 Service 對外暴露服務。OpenELB 項目最初由 KubeSphere 社區 發起,目前已作為 CNCF 沙箱項目 加入 CNCF 基金會,由 OpenELB 開源社區維護與支持。
與MetalLB類似,OpenELB也擁有兩種主要工作模式:Layer2模式和BGP模式。OpenELB的BGP模式目前暫不支持IPv6。
-
layer2 Mode
-
BGP Mode
準備k8s的環境
千云物流測試環境部署使用openelb部署.
所需要的軟件&版本 | 對應依賴軟件版本 |
---|---|
linux [CentOS] | 7.9.2009 |
kubernetes | v1.22.12 |
docker [20.10.8] | 20.10.8 |
openelb | kubesphere/openelb:v0.5.1 |
準備Layer2 Mode配置
- 配置ARP參數
部署Layer2模式需要把k8s集群中的ipvs配置打開strictARP,
strict ARP configure arp_ignore and arp_announce to avoid answering ARP queries from kube-ipvs0 interface
# 查看kube-proxy中的strictARP配置
$ kubectl get configmap -n kube-system kube-proxy -o yaml | grep strictARP
#strictARP: false# 手動修改strictARP配置為true
$ kubectl edit configmap -n kube-system kube-proxy
configmap/kube-proxy edited# 使用命令直接修改并對比不同
$ kubectl get configmap kube-proxy -n kube-system -o yaml | sed -e "s/strictARP: false/strictARP: true/" | kubectl diff -f - -n kube-system# 確認無誤后使用命令直接修改并生效
$ kubectl get configmap kube-proxy -n kube-system -o yaml | sed -e "s/strictARP: false/strictARP: true/" | kubectl apply -f - -n kube-system# 重啟kube-proxy確保配置生效
$ kubectl rollout restart ds kube-proxy -n kube-system# 確認配置生效
$ kubectl get configmap -n kube-system kube-proxy -o yaml | grep strictARPstrictARP: true
開啟之后k8s集群中的kube-proxy會停止響應kube-ipvs0網卡之外的其他網卡的arp請求,而由MetalLB接手處理。
strict ARP開啟之后相當于把將arp_ignore設置為1;并將arp_announce設置為2啟用嚴格的ARP,這個原理和LVS中的DR模式對RS的配置一樣,可以參考之前的文章中的解釋。
網卡配置
#多個網卡,需要指定master節點IP,一個網卡不需要
# kubectl annotate nodes k8s-master01 layer2.openelb.kubesphere.io/v1alpha1="masterip"
創建EIP
接下來我們需要配置loadbalancerIP所在的網段資源,這里我們創建一個Eip對象來進行定義,后面對IP段的管理也是在這里進行。
- 部署eip
apiVersion: network.kubesphere.io/v1alpha2
kind: Eip
metadata:# Eip 對象的名稱。name: layer2-eip
spec:# Eip 對象的地址池address: 10.0.0.122-10.0.0.123# openELB的運行模式,默認為bgpprotocol: layer2# OpenELB 在其上偵聽 ARP/NDP 請求的網卡。該字段僅在protocol設置為時有效layer2。interface: ens160# 指定是否禁用 Eip 對象# false表示可以繼續分配# true表示不再繼續分配disable: false
status:# 指定 Eip 對象中的IP地址是否已用完。occupied: false# 指定 Eip 對象中有多少個 IP 地址已分配給服務。# 直接留空,系統會自動生成usage:# Eip 對象中的 IP 地址總數。poolSize: 2# 指定使用的 IP 地址和使用 IP 地址的服務。服務以Namespace/Service name格式顯示(例如,default/test-svc)。# 直接留空,系統會自動生成used:# Eip 對象中的第一個 IP 地址。firstIP: 10.0.0.122# Eip 對象中的最后一個 IP 地址。lastIP: 10.0.0.123ready: true# 指定IP協議棧是否為 IPv4。目前,OpenELB 僅支持 IPv4,其值只能是true.v4: true
- 檢查eip狀態
kubectl apply -f openelb/openelb-eip.yaml#部署完成后檢查eip的狀態kubectl get eip
部署openelb
這里我們還是使用yaml進行部署,官方把所有部署的資源整合到了一個文件中,我們還是老規矩先下載到本地再進行部署
apiVersion: v1
kind: Namespace
metadata:name: openelb-system
---
apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:annotations:controller-gen.kubebuilder.io/version: v0.4.0creationTimestamp: nullname: bgpconfs.network.kubesphere.io
spec:group: network.kubesphere.ionames:kind: BgpConflistKind: BgpConfListplural: bgpconfssingular: bgpconfscope: Clusterversions:- name: v1alpha1schema:openAPIV3Schema:description: BgpConf is the Schema for the bgpconfs APIproperties:apiVersion:description: 'APIVersion defines the versioned schema of this representationof an object. Servers should convert recognized schemas to the latestinternal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources'type: stringkind:description: 'Kind is a string value representing the REST resource thisobject represents. Servers may infer this from the endpoint the clientsubmits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds'type: stringmetadata:type: objectspec:description: struct for container bgp:config. Configuration parametersrelating to the global BGP router.properties:as:description: original -> bgp:as bgp:as's original type is inet:as-number.Local autonomous system number of the router. Uses the 32-bit as-numbertype from the model in RFC 6991.format: int32type: integerport:description: original -> gobgp:portformat: int32maximum: 65535minimum: 1type: integerrouterID:description: original -> bgp:router-id bgp:router-id's original typeis inet:ipv4-address. Router id of the router, expressed as an 32-bitvalue, IPv4 address.pattern: ^([0-9]{1,3}\.){3}[0-9]{1,3}$type: stringrequired:- as- port- routerIDtype: objectstatus:description: BgpConfStatus defines the observed state of BgpConftype: objecttype: objectserved: truestorage: false- name: v1alpha2schema:openAPIV3Schema:description: BgpConf is the Schema for the bgpconfs APIproperties:apiVersion:description: 'APIVersion defines the versioned schema of this representationof an object. Servers should convert recognized schemas to the latestinternal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources'type: stringkind:description: 'Kind is a string value representing the REST resource thisobject represents. Servers may infer this from the endpoint the clientsubmits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds'type: stringmetadata:type: objectspec:description: Configuration parameters relating to the global BGP router.properties:as:format: int32type: integerasPerRack:additionalProperties:format: int32type: integertype: objectfamilies:items:format: int32type: integertype: arraygracefulRestart:properties:deferralTime:format: int32type: integerenabled:type: booleanhelperOnly:type: booleanlocalRestarting:type: booleanlonglivedEnabled:type: booleanmode:type: stringnotificationEnabled:type: booleanpeerRestartTime:format: int32type: integerpeerRestarting:type: booleanrestartTime:format: int32type: integerstaleRoutesTime:format: int32type: integertype: objectlistenAddresses:items:type: stringtype: arraylistenPort:format: int32type: integerpolicy:type: stringrouterId:type: stringuseMultiplePaths:type: booleantype: objectstatus:description: BgpConfStatus defines the observed state of BgpConfproperties:nodesConfStatus:additionalProperties:properties:as:format: int32type: integerrouterId:type: stringtype: objecttype: objecttype: objecttype: objectserved: truestorage: truesubresources:status: {}
status:acceptedNames:kind: ""plural: ""conditions: []storedVersions: []
---
apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:annotations:controller-gen.kubebuilder.io/version: v0.4.0creationTimestamp: nullname: bgppeers.network.kubesphere.io
spec:group: network.kubesphere.ionames:kind: BgpPeerlistKind: BgpPeerListplural: bgppeerssingular: bgppeerscope: Clusterversions:- name: v1alpha1schema:openAPIV3Schema:description: BgpPeer is the Schema for the bgppeers APIproperties:apiVersion:description: 'APIVersion defines the versioned schema of this representationof an object. Servers should convert recognized schemas to the latestinternal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources'type: stringkind:description: 'Kind is a string value representing the REST resource thisobject represents. Servers may infer this from the endpoint the clientsubmits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds'type: stringmetadata:type: objectspec:properties:addPaths:description: original -> bgp:add-paths Parameters relating to theadvertisement and receipt of multiple paths for a single NLRI (add-paths).properties:sendMax:description: original -> bgp:send-max The maximum number of pathsto advertise to neighbors for a single NLRI.type: integertype: objectconfig:description: original -> bgp:neighbor-address original -> bgp: