k8s学习笔记1-搭建k8s环境（k8s版本1.24.3，ubuntu版本22.04）
介绍
k8s的版本在1.24版本开始，Kubernetes 正式移除对 Dockershim 的支持，Kubernetes1.24 之后，如还想继续在k8s中使用docker，需要自行安装cri-dockerd 组件或者containerd组件，下面的步骤，经过反复测试很多次，步骤应该很稳。
之前我写的一篇关于k8s安装的博客，那个主要是针对k8s版本为1.23.6的，对于1.24及其以上的版本，那个安装博客，不适用
一.环境说明
| 机器名称 | 系统版本 | ip地址 | k8s版本 | docker版本 |
|---|---|---|---|---|
| k8s-master1 | ubuntu 22.04 | 192.168.100.240 | 1.24.3 | 20.10.12 |
| k8s-node1 | ubuntu 22.04 | 192.168.100.243 | 1.24.3 | 20.10.12 |
| k8s-node2 | ubuntu 22.04 | 192.168.100.244 | 1.24.3 | 20.10.12 |
备注说明：
1.这3个机器均是在virtualbox上的虚拟机，虚拟机的网络模式均为bridge adapter模式1
2.虚拟机开机之后，需要执行命令：apt-get update && apt-get upgrade -y && apt install net-tools openssh-server -y
3.修改root用户密码,命令为：passwd root
4.修改ssh配置文件/etc/ssh/sshd_config中的“#PermitRootLogin prohibit-password”，修改为“PermitRootLogin yes”，然后执行命令systemctl restart ssh,使其能够使用root账户，进行ssh登录
5.使用命令sudo apt install docker.io -y安装docker
二.k8s主节点安装
a.环境准备
下面脚本，基本上都是固定格式，后面需要更改的地方是ip和主机名称，需要更改一下，其他均不变
#!/bin/bash
echo "--------------------------------------------------------------1.close firewall---------------------------------------------------------------------"
ufw disable
echo "--------------------------------------------------------------2.close swap-------------------------------------------------------------------------"
#修改swap可以参考链接：https://blog.csdn.net/weixin_42599091/article/details/107164366
#临时关闭
swapoff -a
#永久关闭,这个需要重启生效
sed -i 's#\/swap.img#\#\/swap.img#g' /etc/fstab
echo "--------------------------------------------------------------3.allow iptables bridge flow---------------------------------------------------------"
#参考kubadm官网:https://kubernetes.io/zh/docs/setup/production-environment/tools/kubeadm/install-kubeadm/
cat <<EOF | sudo tee /etc/modules-load.d/k8s.conf
br_netfilter
EOF
cat <<EOF | sudo tee /etc/sysctl.d/k8s.conf
net.bridge.bridge-nf-call-ip6tables = 11
net.bridge.bridge-nf-call-iptables = 1
net.ipv4.ip_forward = 1
EOF
sudo sysctl --system
echo "--------------------------------------------------------------4.modify docker cgroup---------------------------------------------------------------"
#将docker的cgroup修改为systemd的参考链接：https://www.jianshu.com/p/8a62750c0eef
sudo mkdir /etc/docker
cat <<EOF | sudo tee /etc/docker/daemon.json
{
"exec-opts": ["native.cgroupdriver=systemd"],
"log-driver": "json-file",
"log-opts": {
"max-size": "100m"
},
"storage-driver": "overlay2"
}
EOF
sudo systemctl enable docker
sudo systemctl daemon-reload
sudo systemctl restart docker
echo "--------------------------------------------------------------5.add hostname ip----------------------------------------------------------------"
#hosts文件 域名通信
echo 192.168.100.240 k8s-master1 >> /etc/hosts
echo 192.168.100.243 k8s-node1 >> /etc/hosts
echo 192.168.100.244 k8s-node2 >> /etc/hosts
echo "--------------------------------------------------------------6.add k8s source list----------------------------------------------------------------"
#参考链接https://blog.csdn.net/uucckk/article/details/105193431
curl https://mirrors.aliyun.com/kubernetes/apt/doc/apt-key.gpg | sudo apt-key add
echo "deb https://mirrors.aliyun.com/kubernetes/apt kubernetes-xenial main" >> /etc/apt/sources.list
echo "-----------------------------------------------------------7.install k8s apt packages------------------------------------------------------------"
#参考kubadm官网（同步骤3）:https://kubernetes.io/zh/docs/setup/production-environment/tools/kubeadm/install-kubeadm/
sudo apt-get update
sudo apt-get upgrade -y
sudo apt-get install -y apt-transport-https ca-certificates curl
echo "-----------------------------------------------------------8.install kubelet kubeadm kubectl-----------------------------------------------------"
#参考kubadm官网（同步骤3）:https://kubernetes.io/zh/docs/setup/production-environment/tools/kubeadm/install-kubeadm/
sudo apt install kubeadm
sudo apt install kubectl
sudo apt install kubelet
sudo apt-mark hold kubelet kubeadm kubectl
b.安装和配置cri-docker
1.下载最新版cri-docker的网址是：https://github.com/Mirantis/cri-dockerd/tags
(下载很慢，我下载后，分享在百度网盘上：链接: https://pan.baidu.com/s/1MXGVqKyHQLvGX-9D7zewiA 提取码: aaaa )
tar zxf cri-dockerd-0.2.3.amd64.tgz
cp cri-dockerd/cri-dockerd /usr/bin/
2.创建cri-docker启动文件
启动文件从下面链接找到。
https://github.com/Mirantis/cri-dockerd/tree/master/packaging/systemd/cri-docker.service
[root@k8s-master1 ~]# cat /usr/lib/systemd/system/cri-docker.service
[Unit]
Description=CRI Interface for Docker Application Container Engine
Documentation=https://docs.mirantis.com
After=network-online.target firewalld.service docker.service
Wants=network-online.target
Requires=cri-docker.socket
[Service]
Type=notify
ExecStart=/usr/bin/cri-dockerd --network-plugin=cni --pod-infra-container-image=registry.aliyuncs.com/google_containers/pause:3.7
ExecReload=/bin/kill -s HUP $MAINPID
TimeoutSec=0
RestartSec=2
Restart=always
StartLimitBurst=3
StartLimitInterval=60s
LimitNOFILE=infinity
LimitNPROC=infinity
LimitCORE=infinity
TasksMax=infinity
Delegate=yes
KillMode=process
[Install]
WantedBy=multi-user.target
这里/usr/bin/cri-dockerd一定要加上参数
–pod-infra-container-image=registry.aliyuncs.com/google_containers/pause:3.7
其他无须改变
3.创建启动文件，直接从链接中https://github.com/Mirantis/cri-dockerd/blob/master/packaging/systemd/cri-docker.socket，获取就行
[root@k8s-master1 ~]# cat /usr/lib/systemd/system/cri-docker.socket
[Unit]
Description=CRI Docker Socket for the API
PartOf=cri-docker.service
[Socket]
ListenStream=%t/cri-dockerd.sock
SocketMode=0660
SocketUser=root
SocketGroup=docker
[Install]
WantedBy=sockets.target
4.启动cri-docker并设置开机自动启动
[root@k8s-master1 ~]# systemctl daemon-reload ; systemctl enable cri-docker --now
c.初始化master节点
kubeadm init --image-repository registry.aliyuncs.com/google_containers --pod-network-cidr=10.244.0.0/16 --cri-socket unix://var/run/cri-dockerd.sock --ignore-preflight-errors=NumCPU
root@k8s-master1:# kubeadm init --image-repository registry.aliyuncs.com/google_containers --pod-network-cidr=10.244.0.0/16 --cri-socket unix://var/run/cri-dockerd.sock --ignore-preflight-errors=NumCPU
[init] Using Kubernetes version: v1.24.3
[preflight] Running pre-flight checks
[WARNING NumCPU]: the number of available CPUs 1 is less than the required 2
[WARNING SystemVerification]: missing optional cgroups: blkio
[preflight] Pulling images required for setting up a Kubernetes cluster
[preflight] This might take a minute or two, depending on the speed of your internet connection
[preflight] You can also perform this action in beforehand using 'kubeadm config images pull'
[certs] Using certificateDir folder "/etc/kubernetes/pki"
[certs] Generating "ca" certificate and key
[certs] Generating "apiserver" certificate and key
[certs] apiserver serving cert is signed for DNS names [k8s-master1 kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local] and IPs [10.96.0.1 192.168.100.240]
[certs] Generating "apiserver-kubelet-client" certificate and key
[certs] Generating "front-proxy-ca" certificate and key
[certs] Generating "front-proxy-client" certificate and key
[certs] Generating "etcd/ca" certificate and key
[certs] Generating "etcd/server" certificate and key
[certs] etcd/server serving cert is signed for DNS names [k8s-master1 localhost] and IPs [192.168.100.240 127.0.0.1 ::1]
[certs] Generating "etcd/peer" certificate and key
[certs] etcd/peer serving cert is signed for DNS names [k8s-master1 localhost] and IPs [192.168.100.240 127.0.0.1 ::1]
[certs] Generating "etcd/healthcheck-client" certificate and key
[certs] Generating "apiserver-etcd-client" certificate and key
[certs] Generating "sa" key and public key
[kubeconfig] Using kubeconfig folder "/etc/kubernetes"
[kubeconfig] Writing "admin.conf" kubeconfig file
[kubeconfig] Writing "kubelet.conf" kubeconfig file
[kubeconfig] Writing "controller-manager.conf" kubeconfig file
[kubeconfig] Writing "scheduler.conf" kubeconfig file
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Starting the kubelet
[control-plane] Using manifest folder "/etc/kubernetes/manifests"
[control-plane] Creating static Pod manifest for "kube-apiserver"
[control-plane] Creating static Pod manifest for "kube-controller-manager"
[control-plane] Creating static Pod manifest for "kube-scheduler"
[etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests"
[wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s
[apiclient] All control plane components are healthy after 8.517858 seconds
[upload-config] Storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace
[kubelet] Creating a ConfigMap "kubelet-config" in namespace kube-system with the configuration for the kubelets in the cluster
[upload-certs] Skipping phase. Please see --upload-certs
[mark-control-plane] Marking the node k8s-master1 as control-plane by adding the labels: [node-role.kubernetes.io/control-plane node.kubernetes.io/exclude-from-external-load-balancers]
[mark-control-plane] Marking the node k8s-master1 as control-plane by adding the taints [node-role.kubernetes.io/master:NoSchedule node-role.kubernetes.io/control-plane:NoSchedule]
[bootstrap-token] Using token: 6fiwtu.wsx6zl8ov09uu30c
[bootstrap-token] Configuring bootstrap tokens, cluster-info ConfigMap, RBAC Roles
[bootstrap-token] Configured RBAC rules to allow Node Bootstrap tokens to get nodes
[bootstrap-token] Configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials
[bootstrap-token] Configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token
[bootstrap-token] Configured RBAC rules to allow certificate rotation for all node client certificates in the cluster
[bootstrap-token] Creating the "cluster-info" ConfigMap in the "kube-public" namespace
[kubelet-finalize] Updating "/etc/kubernetes/kubelet.conf" to point to a rotatable kubelet client certificate and key
[addons] Applied essential addon: CoreDNS
[addons] Applied essential addon: kube-proxy
Your Kubernetes control-plane has initialized successfully!
To start using your cluster, you need to run the following as a regular user:
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config
Alternatively, if you are the root user, you can run:
export KUBECONFIG=/etc/kubernetes/admin.conf
You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
https://kubernetes.io/docs/concepts/cluster-administration/addons/
Then you can join any number of worker nodes by running the following on each as root:
kubeadm join 192.168.100.240:6443 --token 6fiwtu.wsx6zl8ov09uu30c \
--discovery-token-ca-cert-hash sha256:0ecc0a0ab5b0f1e3f0896329e8ae714b1325c86843f388815138fd15b5e23720
集群初始化成功后，运行如下命令：
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config
export KUBECONFIG=/etc/kubernetes/admin.conf
d.命令kubectl补全
具体可以参考链接：https://blog.csdn.net/weixin_45552105/article/details/118111521
apt install -y bash-completion
source /usr/share/bash-completion/bash_completion
source <(kubectl completion bash)
echo "source <(kubectl completion bash)" >> ~/.bashrc
e.安装网络
关于网络，使用的是flannel，主要参考的是如下这篇文章：https://blog.csdn.net/weixin_43656190/article/details/117604612
1.先将域名解析写到hosts文件里
echo 52.74.223.119 github.com >> /etc/hosts
echo 192.30.253.119 gist.github.com >> /etc/hosts
echo 54.169.195.247 api.github.com >> /etc/hosts
echo 185.199.111.153 assets-cdn.github.com >> /etc/hosts
echo 151.101.64.133 raw.githubusercontent.com >> /etc/hosts
echo 151.101.108.133 user-images.githubusercontent.com >> /etc/hosts
echo 151.101.76.133 gist.githubusercontent.com >> /etc/hosts
echo 151.101.76.133 cloud.githubusercontent.com >> /etc/hosts
echo 151.101.76.133 camo.githubusercontent.com >> /etc/hosts
echo 151.101.76.133 avatars0.githubusercontent.com >> /etc/hosts
echo 151.101.76.133 avatars1.githubusercontent.com >> /etc/hosts
echo 151.101.76.133 avatars2.githubusercontent.com >> /etc/hosts
echo 151.101.76.133 avatars3.githubusercontent.com >> /etc/hosts
echo 151.101.76.133 avatars4.githubusercontent.com >> /etc/hosts
echo 151.101.76.133 avatars5.githubusercontent.com >> /etc/hosts
echo 151.101.76.133 avatars6.githubusercontent.com >> /etc/hosts
echo 151.101.76.133 avatars7.githubusercontent.com >> /etc/hosts
echo 151.101.76.133 avatars8.githubusercontent.com >> /etc/hosts
2.下载kube-flannel.yml文件(有时也会出现下载超级慢的情况，可以在我的百度网盘下载，链接如下: https://pan.baidu.com/s/1UIWluneXcrig3xfqsd-iQA 提取码: aaaa )
wget --no-check-certificate https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml
3.部署flannel文件，如果要是将vxlan模式更改为host-gw方式，需要提前修改kube-flannel.yml文件
kubectl apply -f kube-flannel.yml
应用成功之后，可以看到coredns这类的pod都处于running状态
root@k8s-master1:~# kubectl get pods -n kube-system
NAME READY STATUS RESTARTS AGE
coredns-74586cf9b6-5j424 1/1 Running 0 2m7s
coredns-74586cf9b6-gw9l2 1/1 Running 0 2m7s
etcd-k8s-master1 1/1 Running 0 2m19s
kube-apiserver-k8s-master1 1/1 Running 0 2m19s
kube-controller-manager-k8s-master1 1/1 Running 0 2m22s
kube-proxy-kst4h 1/1 Running 0 2m8s
kube-scheduler-k8s-master1 1/1 Running 0 2m21s
到此master节点已经安装完成
三.k8s子节点安装
关于子节点的安装，需要执行主节点的a和b步骤，然后再加上kubeadm join命令，就可以完成子节点的安装
a.环境准备
下面脚本，基本上都是固定格式，后面需要更改的地方是ip和主机名称，需要更改一下，其他均不变
#!/bin/bash
echo "--------------------------------------------------------------1.close firewall---------------------------------------------------------------------"
ufw disable
echo "--------------------------------------------------------------2.close swap-------------------------------------------------------------------------"
#修改swap可以参考链接：https://blog.csdn.net/weixin_42599091/article/details/107164366
#临时关闭
swapoff -a
#永久关闭,这个需要重启生效
sed -i 's#\/swap.img#\#\/swap.img#g' /etc/fstab
echo "--------------------------------------------------------------3.allow iptables bridge flow---------------------------------------------------------"
#参考kubadm官网:https://kubernetes.io/zh/docs/setup/production-environment/tools/kubeadm/install-kubeadm/
cat <<EOF | sudo tee /etc/modules-load.d/k8s.conf
br_netfilter
EOF
cat <<EOF | sudo tee /etc/sysctl.d/k8s.conf
net.bridge.bridge-nf-call-ip6tables = 11
net.bridge.bridge-nf-call-iptables = 1
net.ipv4.ip_forward = 1
EOF
sudo sysctl --system
echo "--------------------------------------------------------------4.modify docker cgroup------------步骤---------------------------------------------------"
#将docker的cgroup修改为systemd的参考链接：https://www.jianshu.com/p/8a62750c0eef
sudo mkdir /etc/docker
cat <<EOF | sudo tee /etc/docker/daemon.json
{
"exec-opts": ["native.cgroupdriver=systemd"],
"log-driver": "json-file",
"log-opts": {
"max-size": "100m"
},
"storage-driver": "overlay2"
}
EOF
sudo systemctl enable docker
sudo systemctl daemon-reload
sudo systemctl restart docker
echo "--------------------------------------------------------------5.add hostname ip----------------------------------------------------------------"
#hosts文件 域名通信
echo 192.168.100.240 k8s-master1 >> /etc/hosts
echo 192.168.100.243 k8s-node1 >> /etc/hosts
echo 192.168.100.244 k8s-node2 >> /etc/hosts
echo "--------------------------------------------------------------6.add k8s source list----------------------------------------------------------------"
#参考链接https://blog.csdn.net/uucckk/article/details/105193431
curl https://mirrors.aliyun.com/kubernetes/apt/doc/apt-key.gpg | sudo apt-key add
echo "deb https://mirrors.aliyun.com/kubernetes/apt kubernetes-xenial main" >> /etc/apt/sources.list
echo "-----------------------------------------------------------7.install k8s apt packages------------------------------------------------------------"
#参考kubadm官网（同步骤3）:https://kubernetes.io/zh/docs/setup/production-environment/tools/kubeadm/install-kubeadm/
sudo apt-get update
sudo apt-get upgrade -y
sudo apt-get install -y apt-transport-https ca-certificates curl
echo "-----------------------------------------------------------8.install kubelet kubeadm kubectl-----------------------------------------------------"
#参考kubadm官网（同步骤3）:https://kubernetes.io/zh/docs/setup/production-environment/tools/kubeadm/install-kubeadm/
sudo apt install kubeadm
sudo apt install kubectl
sudo apt install kubelet
sudo apt-mark hold kubelet kubeadm kubectl
b.安装和配置cri-docker
1.下载最新版cri-docker的网址是：https://github.com/Mirantis/cri-dockerd/tags
(下载很慢，我下载后，分享在百度网盘上：链接: https://pan.baidu.com/s/1MXGVqKyHQLvGX-9D7zewiA 提取码: aaaa )
tar zxf cri-dockerd-0.2.3.amd64.tgz
cp cri-dockerd/cri-dockerd /usr/bin/
2.创建cri-docker启动文件
启动文件从下面链接找到。
https://github.com/Mirantis/cri-dockerd/tree/master/packaging/systemd/cri-docker.service
[root@k8s-master1 ~]# cat /usr/lib/systemd/system/cri-docker.service
[Unit]
Description=CRI Interface for Docker Application Container Engine
Documentation=https://docs.mirantis.com
After=network-online.target firewalld.service docker.service
Wants=network-online.target
Requires=cri-docker.socket
[Service]
Type=notify
ExecStart=/usr/bin/cri-dockerd --network-plugin=cni --pod-infra-container-image=registry.aliyuncs.com/google_containers/pause:3.7
ExecReload=/bin/kill -s HUP $MAINPID
TimeoutSec=0
RestartSec=2
Restart=always
StartLimitBurst=3
StartLimitInterval=60s
LimitNOFILE=infinity
LimitNPROC=infinity
LimitCORE=infinity
TasksMax=infinity
Delegate=yes
KillMode=process
[Install]
WantedBy=multi-user.target
这里/usr/bin/cri-dockerd一定要加上参数
–pod-infra-container-image=registry.aliyuncs.com/google_containers/pause:3.7
其他无须改变
3.创建启动文件，直接从链接中https://github.com/Mirantis/cri-dockerd/blob/master/packaging/systemd/cri-docker.socket，获取就行
[root@k8s-master1 ~]# cat /usr/lib/systemd/system/cri-docker.socket
[Unit]
Description=CRI Docker Socket for the API
PartOf=cri-docker.service
[Socket]
ListenStream=%t/cri-dockerd.sock
SocketMode=0660
SocketUser=root
SocketGroup=docker
[Install]
WantedBy=sockets.target
4.启动cri-docker并设置开机自动启动
[root@k8s-master1 ~]# systemctl daemon-reload ; systemctl enable cri-docker --now
c.加入集群
方式1： 通过主节点创建的集群时，生成命令进行加入集群
kubeadm join 192.168.100.240:6443 --token 6fiwtu.wsx6zl8ov09uu30c \
--discovery-token-ca-cert-hash sha256:0ecc0a0ab5b0f1e3f0896329e8ae714b1325c86843f388815138fd15b5e23720 --cri-socket unix://var/run/cri-dockerd.sock
方式2： 通过如下命令，获取加入集群的命令（具体可以参考网址：https://www.csdn.net/tags/MtTaEg0sNjE4NDg1LWJsb2cO0O0O.html）
root@k8s-master1:~# kubeadm token create --print-join-command
kubeadm join 192.168.100.240:6443 --token xaib7u.7s0fhn8avqts8768 --discovery-token-ca-cert-hash sha256:0ecc0a0ab5b0f1e3f0896329e8ae714b1325c86843f388815138fd15b5e23720
运行该加入命令，尾部需要增加参数：–cri-socket unix://var/run/cri-dockerd.sock
否则会报如下错误：
root@k8s-node1:~# kubeadm join 192.168.100.240:6443 --token euhl30.ijrfih6gj983cvmr \
--discovery-token-ca-cert-hash sha256:deaceb37f98447bcb023f292b1990cdf7ce5d8402cf2a61276f33450c96238b2
Found multiple CRI endpoints on the host. Please define which one do you wish to use by setting the 'criSocket' field in the kubeadm configuration file: unix:///var/run/containerd/containerd.sock, unix:///var/run/cri-dockerd.sock
To see the stack trace of this error execute with --v=5 or higher
四.验证
1.获取节点
root@k8s-master1:~# kubectl get nodes
NAME STATUS ROLES AGE VERSION
k8s-master1 Ready control-plane 14m v1.24.3
k8s-node1 Ready <none> 2m7s v1.24.3
k8s-node2 Ready <none> 97s v1.24.3
2.获取coredns的pods，均能正常跑起来
root@k8s-master1:~# kubectl get pods -n kube-system
NAME READY STATUS RESTARTS AGE
coredns-74586cf9b6-476cz 1/1 Running 0 14m
coredns-74586cf9b6-trxrm 1/1 Running 0 14m
etcd-k8s-master1 1/1 Running 0 15m
kube-apiserver-k8s-master1 1/1 Running 0 15m
kube-controller-manager-k8s-master1 1/1 Running 0 15m
kube-proxy-8gjdf 1/1 Running 0 114s
kube-proxy-h5c7j 1/1 Running 0 14m
kube-proxy-kwf78 1/1 Running 0 2m24s
kube-scheduler-k8s-master1 1/1 Running 0 15m
3.获取flannel的pods
root@k8s-master1:~# kubectl get pods -n kube-flannel
NAME READY STATUS RESTARTS AGE
kube-flannel-ds-dpj2z 1/1 Running 0 8m58s
kube-flannel-ds-gqx2g 1/1 Running 0 2m8s
kube-flannel-ds-k95d8 1/1 Running 0 2m38s
五.参考资料和说明
1.k8s安装
https://blog.csdn.net/uucckk/article/details/105193431
2.k8s安装（这篇是我自己之前写的）
https://blog.csdn.net/weixin_43501172/article/details/124341680
3.假如忘记join命令，可以参考下面的链接
https://www.csdn.net/tags/MtTaEg0sNjE4NDg1LWJsb2cO0O0O.html
4.kubectl命令补全
https://blog.csdn.net/weixin_45552105/article/details/118111521
版权声明：本文为weixin_43501172原创文章，遵循CC 4.0 BY-SA版权协议，转载请附上原文出处链接和本声明。