kubeadm+haproxy+keepalived部署高可用k8s集群-版本k8s1.20.4—详细文档

kubeadm+haproxy+keepalived部署高可用k8s集群-版本k8s1.20.4—详细文档

相关配套软件包网盘下载链接如下：
网盘地址： https://url28.ctfile.com/f/37115828-599516373-25f42e?p=4907
访问密码：4907

本人会经常更新运维相关技术文档，如有兴趣，可以关注我博客，欢迎互动分享

kubeadm初始化高可用k8s1.20.4集群
1.所有节点修改主机名和配置host解析
#cat /etc/hosts
192.168.27.128 m1
192.168.27.129 m2
192.168.27.130 m3
192.168.27.131 n1
2.所有节点做时间同步：
#ntpdate time.windows.com
3.所有节点关闭防火墙
#systemctl stop firewalld
#systemctl disable firewalld
4.所有节点关闭selinux
#sed -i ‘s/enforcing/disabled/’ /etc/selinux/config
#setenforce 0
5.所有节点关闭swap
#swapoff -a
#sed -i ‘/swap/s/^(.*)$/#\1/g’ /etc/fstab
6.所有节点配置将桥接的IPv4流量传递到iptables的链
#cat >/etc/sysctl.d/k8s.conf << EOF
net.bridge.bridge-nf-call-ip6tables =1
net.bridge.bridge-nf-call-iptables =1
EOF
#sysctl --system #刷新生效
7.所有节点安装docker
#wget https://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo -O /etc/yum.repos.d/docker-ce.repo
#yum -y install docker-ce
#systemctl enable docker
#systemctl start docker
#mkdir /data
#vi /etc/docker/daemon.json #配置镜像下载加速器
{
“graph”: “/data/docker”,
“insecure-registries”:[“https://b9pmyelo.mirror.aliyuncs.com”]
}
#systemctl restart docker
8.所有节点安装kubeadm、kubelet、kubectl
添加阿里云YUM软件源:
#vi /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64
enabled=1
gpgcheck=0
#repo_gpgcheck=0
#gpgkey=https://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg https://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg
#yum install -y kubelet-1.20.4 kubeadm-1.20.4 kubectl-1.20.4
#systemctl enable kubelet #开启开机自启,不用启动,初始化时会自动启动
9.三个master节点部署keepalived - apiserver高可用
Master1节点：
[root@m1 ~]# yum install -y keepalived
[root@m1 ~]# mkdir -p /etc/keepalived
[root@m1 ~]# vim /etc/keepalived/keepalived.conf
! Configuration File for keepalived
global_defs {
router_id keepalive-master
}
vrrp_script check_apiserver {
#检测脚本路径
script “/etc/keepalived/check-apiserver.sh”
#多少秒检测一次
interval 3
#失败的话权重-2
weight -2
}
vrrp_instance VI-kube-master {
state MASTER #定义节点角色
interface ens33 #网卡名称
virtual_router_id 51
priority 100
advert_int 1
authentication {
auth_type PASS
auth_pass 1111
}
virtual_ipaddress {
#自定义虚拟ip
192.168.27.100
}
track_script {
check_apiserver
}
}
[root@m1 ~]# cat /etc/keepalived/check-apiserver.sh
#!/bin/sh
netstat -ntlp |grep 6443 || exit 1
[root@m1 ~]# chmod +x /etc/keepalived/check-apiserver.sh
Master2节点：
[root@m2 ~]# yum install -y keepalived
[root@m2 ~]# mkdir -p /etc/keepalived
[root@m2 ~]# vim /etc/keepalived/keepalived.conf
! Configuration File for keepalived
global_defs {
router_id keepalive-backup
}
vrrp_script check_apiserver {
#检测脚本路径
script “/etc/keepalived/check-apiserver.sh”
#多少秒检测一次
interval 3
#失败的话权重-2
weight -2
}
vrrp_instance VI-kube-master {
state BACKUP #定义节点角色
interface ens33 #网卡名称
virtual_router_id 51
priority 95
advert_int 1
authentication {
auth_type PASS
auth_pass 1111
}
virtual_ipaddress {
#自定义虚拟ip
192.168.27.100
}
track_script {
check_apiserver
}
}
[root@m2 ~]# cat /etc/keepalived/check-apiserver.sh
#!/bin/sh
netstat -ntlp |grep 6443 || exit 1
[root@m2 ~]# chmod +x /etc/keepalived/check-apiserver.sh
Master3节点：
[root@m3 ~]# yum install -y keepalived
[root@m3 ~]# mkdir -p /etc/keepalived
[root@m3 ~]# vim /etc/keepalived/keepalived.conf
! Configuration File for keepalived
global_defs {
router_id keepalive-backup
}
vrrp_script check_apiserver {
#检测脚本路径
script “/etc/keepalived/check-apiserver.sh”
#多少秒检测一次
interval 3
#失败的话权重-2
weight -2
}
vrrp_instance VI-kube-master {
state MASTER #定义节点角色
interface ens33 #网卡名称
virtual_router_id 51
priority 80
advert_int 1
authentication {
auth_type PASS
auth_pass 1111
}
virtual_ipaddress {
#自定义虚拟ip
192.168.27.100
}
track_script {
check_apiserver
}
}
[root@m3 ~]# cat /etc/keepalived/check-apiserver.sh
#!/bin/sh
netstat -ntlp |grep 6443 || exit 1
[root@m3 ~]# chmod +x /etc/keepalived/check-apiserver.sh
完成上述步骤后，启动keepalived：
分别在master和2个backup上启动keepalived服务：
[root@m1 ~]# systemctl enable keepalived && systemctl start keepalived
[root@m2 ~]# systemctl enable keepalived && systemctl start keepalived
[root@m3 ~]# systemctl enable keepalived && systemctl start keepalived
[root@m1 ~]# ip a |grep 27.100
inet 192.168.27.100/32 scope global ens33
[root@m2 ~]# ip a |grep 27.100
空
[root@m3 ~]# ip a |grep 27.100
空
10.部署第一个k8s主节点
[root@m1 ~]# vi kubeadm-config.yaml
apiVersion: kubeadm.k8s.io/v1beta2
kind: ClusterConfiguration
kubernetesVersion: v1.20.4
#指定控制面板的访问端点,这里的ip为vip:
controlPlaneEndpoint: “192.168.27.100:6443”
imageRepository: registry.cn-hangzhou.aliyuncs.com/google_containers
networking:
dnsDomain: cluster.local
podSubnet: 10.244.0.0/16
serviceSubnet: 10.1.0.0/16
[root@m1 ~]# kubeadm init --config=kubeadm-config.yaml --upload-certs
…………
Your Kubernetes control-plane has initialized successfully!
To start using your cluster, you need to run the following as a regular user:
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown ( i d − u ) : (id -u):(id−u):(id -g) $HOME/.kube/config
Alternatively, if you are the root user, you can run:
export KUBECONFIG=/etc/kubernetes/admin.conf
You should now deploy a pod network to the cluster.
Run “kubectl apply -f [podnetwork].yaml” with one of the options listed at:
https://kubernetes.io/docs/concepts/cluster-administration/addons/
You can now join any number of the control-plane node running the following command on each as root:
kubeadm join 192.168.27.100:6443 --token 9mk3al.2vxwgxvgs8xlti22
–discovery-token-ca-cert-hash sha256:880401041d1d1835248bd394a4e58d43f518835da9ec42423476d184a38c8908
–control-plane --certificate-key e2e5c073da9fb13fabd6794dd25705256eb4b53ff4f9ce74fa13b0436041fd69
Please note that the certificate-key gives access to cluster sensitive data, keep it secret!
As a safeguard, uploaded-certs will be deleted in two hours; If necessary, you can use
“kubeadm init phase upload-certs --upload-certs” to reload certs afterward.
Then you can join any number of worker nodes by running the following on each as root:
kubeadm join 192.168.27.100:6443 --token 9mk3al.2vxwgxvgs8xlti22
–discovery-token-ca-cert-hash sha256:880401041d1d1835248bd394a4e58d43f518835da9ec42423476d184a38c8908
……………
拷贝一下这里打印出来的两条kubeadm join命令，后面添加其他master节点以及worker节点时需要用到
然后在master节点上执行如下命令拷贝配置文件：
[root@m1 ~]# mkdir -p $HOME/.kube
[root@m1 ~]# sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
[root@m1 ~]# sudo chown ( i d − u ) : (id -u):(id−u):(id -g) $HOME/.kube/config
[root@m1 ~]# kubectl get node
NAME STATUS ROLES AGE VERSION
m1 NotReady control-plane,master 4m1s v1.20.4
11.后面两个master节点加入集群(以master的角色加入集群)(后面两台master节点操作)
Master2节点加入集群：
[root@m2 ~]# kubeadm join 192.168.27.100:6443 --token 9mk3al.2vxwgxvgs8xlti22
–discovery-token-ca-cert-hash sha256:880401041d1d1835248bd394a4e58d43f518835da9ec42423476d184a38c8908
–control-plane --certificate-key e2e5c073da9fb13fabd6794dd25705256eb4b53ff4f9ce74fa13b0436041fd69 #回车
…………
This node has joined the cluster and a new control plane instance was created:

  • Certificate signing request was sent to apiserver and approval was received.
  • The Kubelet was informed of the new secure connection details.
  • Control plane (master) label and taint were applied to the new node.
  • The Kubernetes control plane instances scaled up.
  • A new etcd member was added to the local/stacked etcd cluster.
    To start administering your cluster from this node, you need to run the following as a regular user:
    mkdir -p $HOME/.kube
    sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
    sudo chown ( i d − u ) : (id -u):(id−u):(id -g) $HOME/.kube/config
    Run ‘kubectl get nodes’ to see this node join the cluster.
    [root@m2 ~]# mkdir -p $HOME/.kube
    [root@m2 ~]# sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
    [root@m2 ~]# sudo chown ( i d − u ) : (id -u):(id−u):(id -g) $HOME/.kube/config
    [root@m2 ~]# kubectl get node
    NAME STATUS ROLES AGE VERSION
    m1 NotReady control-plane,master 12m v1.20.4
    m2 NotReady control-plane,master 75s v1.20.4
    master3节点加入集群：
    [root@m3 ~]# kubeadm join 192.168.27.100:6443 --token 9mk3al.2vxwgxvgs8xlti22
    –discovery-token-ca-cert-hash sha256:880401041d1d1835248bd394a4e58d43f518835da9ec42423476d184a38c8908
    –control-plane --certificate-key e2e5c073da9fb13fabd6794dd25705256eb4b53ff4f9ce74fa13b0436041fd69 #回车
    …………
    This node has joined the cluster and a new control plane instance was created:
  • Certificate signing request was sent to apiserver and approval was received.
  • The Kubelet was informed of the new secure connection details.
  • Control plane (master) label and taint were applied to the new node.
  • The Kubernetes control plane instances scaled up.
  • A new etcd member was added to the local/stacked etcd cluster.
    To start administering your cluster from this node, you need to run the following as a regular user:
    mkdir -p $HOME/.kube
    sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
    sudo chown ( i d − u ) : (id -u):(id−u):(id -g) $HOME/.kube/config
    Run ‘kubectl get nodes’ to see this node join the cluster.
    [root@m3 ~]# mkdir -p $HOME/.kube
    [root@m3 ~]# sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
    [root@m3 ~]# sudo chown ( i d − u ) : (id -u):(id−u):(id -g) $HOME/.kube/config
    [root@m3 ~]# kubectl get node
    NAME STATUS ROLES AGE VERSION
    m1 NotReady control-plane,master 12m v1.20.4
    m2 NotReady control-plane,master 5m v1.20.4
    m3 NotReady control-plane,master 75s v1.20.4

12.将node节点加入集群(以node的角色加入集群)(在node节点操作)
[root@n1 ~]# kubeadm join 192.168.27.100:6443 --token 9mk3al.2vxwgxvgs8xlti22
–discovery-token-ca-cert-hash sha256:880401041d1d1835248bd394a4e58d43f518835da9ec42423476d184a38c8908 #回车
………
This node has joined the cluster:

  • Certificate signing request was sent to apiserver and a response was received.
  • The Kubelet was informed of the new secure connection details.
    Run ‘kubectl get nodes’ on the control-plane to see this node join the cluster.
    13.在master1节点上部署flannel网络插件和查看k8s集群节点
    [root@m1 ~]# wget https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml
    [root@m1 ~]# kubectl apply -f kube-flannel.yml
    [root@m1 ~]# kubectl get pod -n kube-system
    NAME READY STATUS RESTARTS AGE
    coredns-54d67798b7-7tkzh 1/1 Running 0 29m
    coredns-54d67798b7-r5zhp 1/1 Running 0 29m
    etcd-m1 1/1 Running 0 29m
    etcd-m2 1/1 Running 0 27m
    etcd-m3 1/1 Running 0 24m
    kube-apiserver-m1 1/1 Running 0 29m
    kube-apiserver-m2 1/1 Running 0 27m
    kube-apiserver-m3 1/1 Running 0 24m
    kube-controller-manager-m1 1/1 Running 1 29m
    kube-controller-manager-m2 1/1 Running 0 27m
    kube-controller-manager-m3 1/1 Running 0 24m
    kube-flannel-ds-4rdx5 1/1 Running 0 20m
    kube-flannel-ds-5p7h6 1/1 Running 0 20m
    kube-flannel-ds-8nkp4 1/1 Running 0 3m16s
    kube-flannel-ds-sbbnn 1/1 Running 0 20m
    kube-proxy-56f4x 1/1 Running 0 3m16s
    kube-proxy-68jv9 1/1 Running 0 29m
    kube-proxy-tklkn 1/1 Running 0 27m
    kube-proxy-zzczj 1/1 Running 0 25m
    kube-scheduler-m1 1/1 Running 1 29m
    kube-scheduler-m2 1/1 Running 0 27m
    kube-scheduler-m3 1/1 Running 0 24m
    在3个master上任意一个节点都能查看集群node节点和管理集群：
    在master1节点查看集群：
    [root@m1 ~]# kubectl get node
    NAME STATUS ROLES AGE VERSION
    m1 Ready control-plane,master 30m v1.20.4
    m2 Ready control-plane,master 27m v1.20.4
    m3 Ready control-plane,master 25m v1.20.4
    n1 Ready 3m56s v1.20.4
    在master2节点查看集群：
    [root@m2 ~]# kubectl get node
    NAME STATUS ROLES AGE VERSION
    m1 Ready control-plane,master 31m v1.20.4
    m2 Ready control-plane,master 29m v1.20.4
    m3 Ready control-plane,master 27m v1.20.4
    n1 Ready 5m17s v1.20.4
    在master3节点查看集群：
    [root@m3 ~]# kubectl get node
    NAME STATUS ROLES AGE VERSION
    m1 Ready control-plane,master 32m v1.20.4
    m2 Ready control-plane,master 29m v1.20.4
    m3 Ready control-plane,master 27m v1.20.4
    n1 Ready 5m36s v1.20.4
    13.模拟在master1节点宕机，看是否能将vip漂移到master2节点（优先级高）
    模拟master1宕机：
    [root@m1 ~]# systemctl stop keepalived
    [root@m1 ~]# ip a |grep 27.100
    空
    [root@m1 ~]# systemctl stop kubelet
    [root@m1 ~]# systemctl stop docker
    查看master2节点接管：
    [root@m2 ~]# ip a |grep 27.100
    inet 192.168.27.100/32 scope global ens33
    [root@m2 ~]# kubectl get node
    NAME STATUS ROLES AGE VERSION
    m1 NotReady control-plane,master 52m v1.20.4
    m2 Ready control-plane,master 49m v1.20.4
    m3 Ready control-plane,master 47m v1.20.4
    n1 Ready 25m v1.20.4

模拟master1开机和恢复正常：
[root@m1 ~]# systemctl start docker
[root@m1 ~]# systemctl start kubelet
[root@m1 ~]# systemctl start keepalived
[root@m1 ~]# ip a |grep 27.100
inet 192.168.27.100/32 scope global ens33
[root@m1 ~]# kubectl get node
NAME STATUS ROLES AGE VERSION
m1 Ready control-plane,master 54m v1.20.4
m2 Ready control-plane,master 51m v1.20.4
m3 Ready control-plane,master 49m v1.20.4
n1 Ready 27m v1.20.4
在master2节点查看：
[root@m2 ~]# kubectl get node
NAME STATUS ROLES AGE VERSION
m1 Ready control-plane,master 54m v1.20.4
m2 Ready control-plane,master 52m v1.20.4
m3 Ready control-plane,master 50m v1.20.4
n1 Ready 28m v1.20.4
补充：
1.部署完毕3master节点后，检测etcd集群状态方法：
检测etcd集群后台节点情况：
[root@m1 ~]# /data/docker/… /bin/etcdctl --endpoints=https://127.0.0.1:2379 --cacert=/etc/kubernetes/pki/etcd/ca.crt --cert=/etc/kubernetes/pki/etcd/server.crt --key=/etc/kubernetes/pki/etcd/server.key member list #回车
3d918932a9fd451, started, m2, https://192.168.27.129:2380, https://192.168.27.129:2379, false
18e561669f73fb0d, started, m1, https://192.168.27.128:2380, https://192.168.27.128:2379, false
3cff0e60326db4ca, started, m3, https://192.168.27.130:2380, https://192.168.27.130:2379, false
检测etcd集群健康状态：
[root@m1 ~]# /data/docker/…/bin/etcdctl --endpoints=https://192.168.27.128:2379,https://192.168.27.129:2379,https://192.168.27.130:2379 --cacert=/etc/kubernetes/pki/etcd/ca.crt --cert=/etc/kubernetes/pki/etcd/server.crt --key=/etc/kubernetes/pki/etcd/server.key endpoint health #回车
https://192.168.27.128:2379 is healthy: successfully committed proposal: took = 9.274132ms
https://192.168.27.129:2379 is healthy: successfully committed proposal: took = 9.307928ms
https://192.168.27.130:2379 is healthy: successfully committed proposal: took = 10.585282ms
查看etcd集群状态参考地址： https://www.jianshu.com/p/bf91aa5a636e

2.解决k8s的kubectl get cs 显示不健康的问题：
参考地址： https://www.cnblogs.com/ltaodream/p/15185039.html
[root@m1 ~]# kubectl get cs
Warning: v1 ComponentStatus is deprecated in v1.19+
NAME STATUS MESSAGE ERROR
scheduler Unhealthy Get “http://127.0.0.1:10251/healthz”: dial tcp 127.0.0.1:10251: connect: connection refused
controller-manager Unhealthy Get “http://127.0.0.1:10252/healthz”: dial tcp 127.0.0.1:10252: connect: connection refused
etcd-0 Healthy {“health”:“true”}
解决：
[root@m1 ~]# vim /etc/kubernetes/manifests/kube-scheduler.yaml
#- --port=0 #将–port=0注释
[root@m1 ~]# vim /etc/kubernetes/manifests/kube-controller-manager.yaml
#- --port=0 #将–port=0注释
[root@m1 ~]# systemctl restart kubelet #重启kubelet服务
查看：
Warning: v1 ComponentStatus is deprecated in v1.19+
NAME STATUS MESSAGE ERROR
scheduler Healthy ok
controller-manager Healthy ok
etcd-0 Healthy {“health”:“true”}

版权声明：本文为qq_34953582原创文章，遵循CC 4.0 BY-SA版权协议，转载请附上原文出处链接和本声明。