主機配置
| 作用 | IP地址 | 操作系統 | 配置 | 關鍵組件 |
|---|---|---|---|---|
| k8s-master | 172.16.1.30 | Rocky Linux release 9 | 4C/4G/50GB | kube-apiserver, etcd,docker |
| k8s-node1 | 172.16.1.31 | Rocky Linux release9 | 4C/4G/50GB | kubelet, kube-proxy,docker |
| k8s-node2 | 172.16.1.32 | Rocky Linux release 9 | 4C/4G/50GB | kubelet, kube-proxy,docker |
| k8s-node3 | 172.16.1.33 | Rocky Linux release 9 | 4C/4G/50GB | kubelet, kube-proxy,docker |
設置IP
方式一：
nmcli connection modify ens160 ipv4.addresses 172.16.1.30/24 ipv4.gateway 172.16.1.1 ipv4.method manual
nmcli connection modify ens160 ipv4.addresses 172.16.1.31/24 ipv4.gateway 172.16.1.1 ipv4.method manual
nmcli connection modify ens160 ipv4.addresses 172.16.1.32/24 ipv4.gateway 172.16.1.1 ipv4.method manual
nmcli connection modify ens160 ipv4.addresses 172.16.1.33/24 ipv4.gateway 172.16.1.1 ipv4.method manualnmcli connection up ens160方式二：
vi /etc/NetworkManager/system-connections/ens160.nmconnection
method=manual## 在IPV4下面修改如下內容
address1=192.168.0.5/24,192.168.0.1## 修改IP，子網掩碼（24是子網掩碼的24位，對應255.255.255.0）， 網關
dns=119.29.29.29;114.114.114.114## 設置DNS服務
may-fail=false
重新加載配置文件
nmcli connection reload ens160.nmconnection
激活配置文件
nmcli connection up ens160
配置YUM源
- 配置yum源
（1）確認文件是否存在且可讀
sudo cat /etc/yum.repos.d/rocky.repo
如果文件不存在或內容為空，重新創建它。（2）重新下載正確的阿里云源文件
sudo rm -f /etc/yum.repos.d/rocky.repo # 刪除舊文件（如果有）
sudo curl -o /etc/yum.repos.d/rocky.repo https://mirrors.aliyun.com/rockylinux/rocky.repo?repo=rocky-9
（3）手動編輯文件（如果下載失敗）
sudo vi /etc/yum.repos.d/rocky.repo
粘貼以下內容（阿里云 Rocky Linux 9 鏡像源）：
[baseos]
name=Rocky Linux $releasever - BaseOS - Aliyun
baseurl=https://mirrors.aliyun.com/rockylinux/$releasever/BaseOS/$basearch/os/
gpgcheck=1
enabled=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-rockyofficial[appstream]
name=Rocky Linux $releasever - AppStream - Aliyun
baseurl=https://mirrors.aliyun.com/rockylinux/$releasever/AppStream/$basearch/os/
gpgcheck=1
enabled=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-rockyofficial[extras]
name=Rocky Linux $releasever - Extras - Aliyun
baseurl=https://mirrors.aliyun.com/rockylinux/$releasever/extras/$basearch/os/
gpgcheck=1
enabled=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-rockyofficial（4）也可以直接替換yum源里的地址
sed -e 's|^mirrorlist=|#mirrorlist=|g' \-e 's|^#baseurl=http://dl.rockylinux.org/$contentdir|baseurl=https://mirrors.aliyun.com/rockylinux|g' \-i.bak \/etc/yum.repos.d/rocky*.repo
- 強制替換變量為 Rocky Linux 9
確保 $releasever 和 $basearch 被正確解析：
sudo sed -i 's/$releasever/9/g' /etc/yum.repos.d/rocky.repo
sudo sed -i 's/$basearch/x86_64/g' /etc/yum.repos.d/rocky.repo # 如果是 x86_64 架構
- 導入 GPG 密鑰
sudo rpm --import https://mirrors.aliyun.com/rockylinux/RPM-GPG-KEY-rockyofficial
-
檢查文件權限和格式
（1）確保文件權限正確sudo chmod 644 /etc/yum.repos.d/rocky.repo
（2）檢查文件格式（避免 UTF-8 BOM 或 Windows 換行符）
```bash
sudo dos2unix /etc/yum.repos.d/rocky.repo # 如果是從 Windows 復制的文件
```清除緩存并重新加載
sudo dnf clean all
sudo dnf makecache
- 驗證倉庫是否啟用
sudo dnf repolist
正常輸出應類似：
text
repo id repo name
baseos Rocky Linux 9 - BaseOS - Aliyun
appstream Rocky Linux 9 - AppStream - Aliyun
extras Rocky Linux 9 - Extras - Aliyun
在 Rocky Linux 9 中啟用并安裝 EPEL Repo。
dnf install epel-release
備份(如有配置其他epel源)并替換為國內鏡像
注意最后這個庫，阿里云沒有對應的鏡像，不要修改它，如果誤改恢復原版源即可
cp /etc/yum.repos.d/epel.repo /etc/yum.repos.d/epel.repo.backup
cp /etc/yum.repos.d/epel-testing.repo /etc/yum.repos.d/epel-testing.repo.backup
cp /etc/yum.repos.d/epel-cisco-openh264.repo /etc/yum.repos.d/epel-cisco-openh264.repo.backup
將 repo 配置中的地址替換為阿里云鏡像站地址
執行下面語句，它會替換epel.repo、eple-testing.repo中的網址，不會修改epel-cisco-openh264.repo，可以正常使用。
sed -e 's!^metalink=!#metalink=!g' \-e 's!^#baseurl=!baseurl=!g' \-e 's!https\?://download\.fedoraproject\.org/pub/epel!https://mirrors.aliyun.com/epel!g' \-e 's!https\?://download\.example/pub/epel!https://mirrors.aliyun.com/epel!g' \-i /etc/yum.repos.d/epel{,-testing}.repo
更新倉庫緩存
dnf clean all
dnf makecache ---生成緩存，安裝軟件更快
每臺機器單獨做
hostnamectl set-hostname k8s-master
hostnamectl set-hostname k8s-node1
hostnamectl set-hostname k8s-node2
hostnamectl set-hostname k8s-node3
設置hosts
cat >> /etc/hosts << EOF
172.16.1.30 k8s-master
172.16.1.31 k8s-node1
172.16.1.32 k8s-node2
172.16.1.33 k8s-node3
EOF
配置免密登錄，只在k8s-master上操作
[root@k8s-master ~]# ssh-keygen -f ~/.ssh/id_rsa -N '' -q
拷貝密鑰到其他3 臺節點
[root@k8s-master ~]# ssh-copy-id k8s-node1
[root@k8s-master ~]# ssh-copy-id k8s-node2
[root@k8s-master ~]# ssh-copy-id k8s-node3
防火墻和SELinux
# 關閉防火墻
systemctl disable --now firewalld
# 禁用SELinux
sed -i '/^SELINUX=/ c SELINUX=disabled' /etc/selinux/config
# 重啟生效所以臨時設置為寬容模式
setenforce 0
時間同步配置
# 安裝時間服務器軟件包
dnf install -y chrony
# 修改同步服務器
sed -i '/^pool/ c pool ntp1.aliyun.com iburst' /etc/chrony.conf
systemctl restart chronyd
systemctl enable chronyd
chronyc sources
配置內核轉發及網橋過濾
# 添加網橋過濾及內核轉發配置文件
cat > /etc/sysctl.d/k8s.conf << EOF
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
vm.swappiness = 0
EOF# 加載br_netfilter模塊
modprobe br_netfilter
使用新添加配置文件生效
sysctl -p /etc/sysctl.d/k8s.conf
關閉swap
查看交換分區情況
# 臨時關閉
swapoff -a
# 永遠關閉swap分區
sed -i 's/.*swap.*/#&/' /etc/fstab
啟用ipvs
cat >> /etc/modules-load.d/ipvs.conf << EOF
br_netfilter
ip_conntrack
ip_vs
ip_vs_lc
ip_vs_wlc
ip_vs_rr
ip_vs_wrr
ip_vs_lblc
ip_vs_lblcr
ip_vs_dh
ip_vs_sh
ip_vs_fo
ip_vs_nq
ip_vs_sed
ip_vs_ftp
ip_vs_sh
nf_conntrack
ip_tables
ip_set
xt_set
ipt_set
ipt_rpfilter
ipt_REJECT
ipip
EOF# 安裝依賴
dnf install ipvsadm ipset sysstat conntrack libseccomp -y
重啟服務
systemctl restart systemd-modules-load.service
查看模塊內容
lsmod | grep -e ip_vs -e nf_conntrack
句柄數最大
# 設置為最大
ulimit -SHn 65535cat >> /etc/security/limits.conf <<EOF
* soft nofile 655360
* hard nofile 131072
* soft nproc 655350
* hard nproc 655350
* seft memlock unlimited
* hard memlock unlimitedd
EOF# 查看修改結果
ulimit -a
系統優化
cat > /etc/sysctl.d/k8s_better.conf << EOF
net.bridge.bridge-nf-call-iptables=1
net.bridge.bridge-nf-call-ip6tables=1
net.ipv4.ip_forward=1
vm.swappiness=0
vm.overcommit_memory=1
vm.panic_on_oom=0
fs.inotify.max_user_instances=8192
fs.inotify.max_user_watches=1048576
fs.file-max=52706963
fs.nr_open=52706963
net.ipv6.conf.all.disable_ipv6=1
net.netfilter.nf_conntrack_max=2310720
EOFmodprobe br_netfilter
lsmod |grep conntrack
modprobe ip_conntrack
sysctl -p /etc/sysctl.d/k8s_better.conf
安裝docker
# Step 1: 安裝依賴
yum install -y yum-utils device-mapper-persistent-data lvm2# Step 2: 添加軟件源信息
yum-config-manager --add-repo https://mirrors.aliyun.com/docker-ce/linux/rhel/docker-ce.repo# Step 3: 安裝Docker-CE
yum -y install docker-ce# docker -v
Docker version 27.5.1, build 9f9e405# 設置國內鏡像加速
mkdir -p /etc/docker/
cat >> /etc/docker/daemon.json << EOF
{"registry-mirrors":["https://p3kgr6db.mirror.aliyuncs.com","https://docker.m.daocloud.io","https://your_id.mirror.aliyuncs.com","https://docker.nju.edu.cn/","https://docker.anyhub.us.kg","https://dockerhub.jobcher.com","https://dockerhub.icu","https://docker.ckyl.me","https://cr.console.aliyun.com"],
"exec-opts": ["native.cgroupdriver=systemd"]
}
EOF# 設置docker開機啟動并啟動
systemctl enable --now docker# 查看docker版本
docker version
安裝cri-dockerd
下載地址：Releases · Mirantis/cri-dockerd (github.com)。
https://github.com/Mirantis/cri-dockerd/releases/download/v0.3.16/cri-dockerd-0.3.16-3.fc35.x86_64.rpm
安裝cri-docker
# 下載rpm包
wget -c https://github.com/Mirantis/cri-dockerd/releases/download/v0.3.16/cri-dockerd-0.3.16-3.fc35.x86_64.rpm
wget -c https://rpmfind.net/linux/almalinux/8.10/BaseOS/x86_64/os/Packages/libcgroup-0.41-19.el8.x86_64.rpm# 安裝rpm包
yum install libcgroup-0.41-19.el8.x86_64.rpm
yum install cri-dockerd-0.3.16-3.fc35.x86_64.rpm
設置cri-docker服務開機自啟
systemctl enable cri-docker
cri-docke設置國內鏡像加速
# 編輯service文件
vim /usr/lib/systemd/system/cri-docker.service文件
修改第10行內容
------------------
ExecStart=/usr/bin/cri-dockerd --pod-infra-container-image=registry.aliyuncs.com/google_containers/pause:3.9 --container-runtime-endpoint fd://
-----------------------------------# 重啟Docker組件
systemctl daemon-reload && systemctl restart docker cri-docker.socket cri-docker # 檢查Docker組件狀態
systemctl status docker cir-docker.socket cri-docker
K8S軟件安裝
# 1、配置kubernetes源
cat <<EOF | tee /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=https://mirrors.aliyun.com/kubernetes-new/core/stable/v1.32/rpm/
enabled=1
gpgcheck=1
gpgkey=https://mirrors.aliyun.com/kubernetes-new/core/stable/v1.32/rpm/repodata/repomd.xml.key
EOF# 2、查看所有可用的版本
yum list kubelet --showduplicates | sort -r |grep 1.32# 3、安裝kubelet、kubeadm、kubectl、kubernetes-cni
yum install -y kubelet kubeadm kubectl kubernetes-cni# 4、配置cgroup
為了實現docker使用的cgroupdriver與kubelet使用的cgroup的一致性，建議修改如下文件內容。
vim /etc/sysconfig/kubelet [3臺全部設置下]
---------------------
KUBELET_EXTRA_ARGS="--cgroup-driver=systemd"
---------------------# 5、設置kubelet為開機自啟動即可，由于沒有生成配置文件，集群初始化后自動啟動
systemctl enable kubelet
K8S集群初始化
# 只在k8s-master節點上操作
[root@localhost ~]# kubeadm config print init-defaults > kubeadm-init.yaml# 編輯kubeadm-init.yaml修改如下配置：
- advertiseAddress：為控制平面地址，（Master主機IP）advertiseAddress: 1.2.3.4
修改為 advertiseAddress: 172.16.1.30- criSocket：為 containerd 的socket 文件地址criSocket: unix:///var/run/containerd/containerd.sock
修改為 criSocket: unix:///var/run/cri-dockerd.sock- name: node 修改node為k8s-mastername: node
修改為 name: k8s-master- imageRepository：阿里云鏡像代理地址，否則拉取鏡像會失敗imageRepository: registry.k8s.io
修改為：imageRepository: registry.aliyuncs.com/google_containers- kubernetesVersion：為k8s版本kubernetesVersion: 1.32.0
修改為：kubernetesVersion: 1.32.6# 文件末尾增加啟用ipvs功能
---
apiVersion: kubeproxy.config.k8s.io/v1alpha1
kind: KubeProxyConfiguration
mode: ipvs# 根據配置文件啟動kubeadm初始化k8s
$ kubeadm init --config=kubeadm-init.yaml --upload-certs --v=6
輸出結果：
Your Kubernetes control-plane has initialized successfully!To start using your cluster, you need to run the following as a regular user:mkdir -p $HOME/.kubesudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/configsudo chown $(id -u):$(id -g) $HOME/.kube/configAlternatively, if you are the root user, you can run:export KUBECONFIG=/etc/kubernetes/admin.confYou should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:https://kubernetes.io/docs/concepts/cluster-administration/addons/Then you can join any number of worker nodes by running the following on each as root:kubeadm join 172.16.1.30:6443 --token abcdef.0123456789abcdef \--discovery-token-ca-cert-hash sha256:9d25c16abfec6ff6832ed2260c6c998d3fa6fedef61529d88520d3038bdbdde5
K8S集群工作節點加入
# 注意：加入集群時需要添加 --cri-socket unix:///var/run/cri-dockerd.sock
kubeadm join 172.16.1.30:6443 --token abcdef.0123456789abcdef \--discovery-token-ca-cert-hash sha256:9d25c16abfec6ff6832ed2260c6c998d3fa6fedef61529d88520d3038bdbdde5 \--cri-socket unix:///var/run/cri-dockerd.sock
K8S集群網絡插件使用
# 下載calico資源清單
wget --no-check-certificate https://raw.githubusercontent.com/projectcalico/calico/v3.28.0/manifests/calico.yaml# 修改calico文件
vim calico.yaml
- name: CALICO_IPV4POOL_CIDRvalue: "10.244.0.0/16"# 可以將鏡像提前拉取下來,如果官網倉庫不可達，可以嘗試手動從quay.io下載鏡像，quay.io是一個公共鏡像倉庫。
docker pull calico/cni:v3.28.0
docker pull calico/node:v3.28.0
docker pull calico/kube-controllers:v3.28.0# 應用calico資源清單
kubectl apply -f calico.yaml
Kubectl命令自動補全
yum -y install bash-completion
source /usr/share/bash-completion/bash_completion
source <(kubectl completion bash)
echo "source <(kubectl completion bash)" >> ~/.bashrc
安裝helm v3.16.3
wget https://get.helm.sh/helm-v3.16.3-linux-amd64.tar.gz
tar xf helm-v3.16.3-linux-amd64.tar.gz
cd linux-amd64/
mv helm /usr/local/bin
helm version
部署動態sc存儲
# k8s-master節點上執行
yum -y install nfs-utils
echo "/nfs/data/ *(insecure,rw,sync,no_root_squash)" > /etc/exports
mkdir -p /nfs/data/
chmod 777 -R /nfs/data/
systemctl enable rpcbind
systemctl enable nfs-server
systemctl start rpcbind
systemctl start nfs-server
exportfs -v
創建nfs-provisioner
apiVersion: v1
kind: ServiceAccount
metadata:name: nfs-client-provisioner # sa名字，nfs-provisioner-deploy里的要對應namespace: kube-system # 命名空間
---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1 # 創建集群規則
metadata:name: nfs-client-provisioner-runner
rules:- apiGroups: [""]resources: ["persistentvolumes"]verbs: ["get", "list", "watch", "create", "delete"]- apiGroups: [""]resources: ["persistentvolumeclaims"]verbs: ["get", "list", "watch", "update"]- apiGroups: ["storage.k8s.io"]resources: ["storageclasses"]verbs: ["get", "list", "watch"]- apiGroups: [""]resources: ["events"]verbs: ["create", "update", "patch"]
---
kind: ClusterRoleBinding # 將服務認證用戶與集群規則進行綁定
apiVersion: rbac.authorization.k8s.io/v1
metadata:name: run-nfs-client-provisioner
subjects:- kind: ServiceAccount # 類型為saname: nfs-client-provisioner # sa的名字一致namespace: kube-system # 和nfs provisioner安裝的namespace一致
roleRef:kind: ClusterRolename: nfs-client-provisioner-runnerapiGroup: rbac.authorization.k8s.io
---
kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:name: leader-locking-nfs-client-provisionernamespace: kube-system # 和nfs provisioner安裝的namespace一致
rules:- apiGroups: [""]resources: ["endpoints"]verbs: ["get", "list", "watch", "create", "update", "patch"]
---
kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:name: leader-locking-nfs-client-provisionernamespace: kube-system # 和nfs provisioner安裝的namespace一致
subjects:- kind: ServiceAccount # 類型為saname: nfs-client-provisioner # sa的名字一致namespace: kube-system # 和nfs provisioner安裝的namespace一致
roleRef:kind: Rolename: leader-locking-nfs-client-provisionerapiGroup: rbac.authorization.k8s.io
---
apiVersion: apps/v1
kind: Deployment
metadata:name: nfs-client-provisionerlabels:app: nfs-client-provisionernamespace: kube-system # 部署在指定ns下
spec:replicas: 1 # 副本數，建議為奇數[1，3，5，7，9]strategy:type: Recreate # 使用重建的升級策略selector:matchLabels:app: nfs-client-provisionertemplate:metadata:labels:app: nfs-client-provisionerspec:serviceAccountName: nfs-client-provisioner # sa名字，這個是在nfs-rbac.yaml里定義containers:- name: nfs-client-provisioner # 容器名字image: k8s.m.daocloud.io/sig-storage/nfs-subdir-external-provisioner:v4.0.2 # 鏡像地址，這里采用私有倉庫。volumeMounts:- name: nfs-client-rootmountPath: /persistentvolumes # 指定容器內掛載的目錄env:- name: PROVISIONER_NAME # 容器內的變量用于指定提供存儲的名稱value: nfsnas # nfs-provisioner的名稱，以后設置的storage class要和這個保持一致- name: NFS_SERVER # 容器內的變量指定nfs服務器對應的目錄value: 172.16.1.254 # NFS服務器的地址- name: NFS_PATH # 容器內的變量指定nfs服務器對應的目錄value: /volume1/服務/K8s-NFS # NFS服務的掛載目錄，如果采用這個nfs動態申請PV,所創建的文件在這個目錄里，一定要給權限，直接777。volumes:- name: nfs-client-root # 賦值卷名字nfs:server: 172.16.1.254 # NFS服務器的地址path: /volume1/服務/K8s-NFS # NFS服務的掛載目錄，一定要給權限，直接777，不服就是干
---
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:name: nfsnasannotations:storageclass.kubernetes.io/is-default-class: "true" # 設為默認存儲類
provisioner: nfsnas # 必須與 Deployment 中 PROVISIONER_NAME 一致
parameters:archiveOnDelete: "false" # "true" 表示刪除 PVC 時歸檔數據（重命名目錄）
