Nginx+keepalived 高可用
1、Nginx安装配置
1.1 安装前工作
首先更新系统软件源，使用以下命令更新系统 -
[root@rhel0 ~]# yum update
有关两个命令的一点解释：
yum -y update- 升级所有包，改变软件设置和系统设置,系统版本内核都升级
yum -y upgrade- 升级所有包，不改变软件设置和系统设置，系统版本升级，内核不改变
依赖包安装
[root@rhel0 src]# yum -y install gcc gcc-c++ autoconf automake libtool make cmake
[root@rhel0 src]# yum -y install zlib zlib-devel openssl openssl-devel pcre-devel
1.2 下载Nginx安装源文件
源码下载，可官网下载地址：http://nginx.org/en/download.html 下载并上传到服务器(这里选择最新稳定版本：nginx-1.22.0)，如下图所示
或直接在服务上执行以下命令下载
[root@rhel0 ~]# cd /usr/local/src
[root@rhel0 src]# wget -c http://nginx.org/download/nginx-1.22.0.tar.gz
解压上面下载的文件
[root@rhel0 src]# tar zxvf nginx-1.22.0.tar.gz
在编译之前还要做一些前期的准备工作，如：依懒包安装，Nginx用户和用户组等。
1.3 新建nginx用户及用户组
使用 root 用户身份登录系统，执行以下命令创建新的用户。
[root@rhel0 src]# groupadd nginx
[root@rhel0 src]# useradd -g nginx -M nginx
useradd命令的-M参数用于不为nginx建立home目录
修改/etc/passwd，使得nginx用户无法bash登陆(nginx用户后面由/bin/bash改为/sbin/nologin)，
[root@rhel0 src]# vi /etc/passwd
然后找到有 nginx 那一行，把它修改为(后面由/bin/bash改为/sbin/nologin)：
nginx:x:1002:1003::/home/nginx:/sbin/nologin
1.4 编译配置、编译、安装
下面我们进入解压的nginx源码目录：/usr/local/src/ 执行以下命令 -
[root@rhel0 ~]# cd /usr/local/src/nginx*
[root@rhel0 nginx-1.22.0]# pwd
/usr/local/src/nginx-1.22.0
[root@rhel0 nginx-1.22.0]#
[root@rhel0 nginx-1.22.0]# ./configure --prefix=/usr/local/nginx \
--pid-path=/usr/local/nginx/run/nginx.pid \
--with-http_ssl_module \
--user=nginx \
--group=nginx \
--with-pcre \
--without-mail_pop3_module \
--without-mail_imap_module \
--without-mail_smtp_module
注意：上面的反斜杠
\表示换行继续。
--prefix=/usr/local/nginx 指定安装到 /usr/local/nginx 目录下。
上面配置完成后，接下来执行编译 -
[root@rhel0 nginx-1.22.0]# make
[root@rhel0 nginx-1.22.0]# make install
... ...
cp conf/nginx.conf '/usr/local/nginx/conf/nginx.conf.default'
test -d '/usr/local/nginx/run' \
|| mkdir -p '/usr/local/nginx/run'
test -d '/usr/local/nginx/logs' \
|| mkdir -p '/usr/local/nginx/logs'
test -d '/usr/local/nginx/html' \
|| cp -R html '/usr/local/nginx'
test -d '/usr/local/nginx/logs' \
|| mkdir -p '/usr/local/nginx/logs'
make[1]: Leaving directory `/usr/local/src/nginx-1.22.0'
[root@rhel0 nginx-1.22.0]#
Shell
上面编译时间跟你的电脑配置相关，所以可能需要一些等待时间。
查看安装后的程序版本：
[root@rhel0 nginx-1.22.0]# /usr/local/nginx/sbin/nginx -v
nginx version: nginx/1.22.0
修改Nginx默认端口(可选)：
[root@rhel0 nginx-1.22.0]# vi /usr/local/nginx/conf/nginx.conf
找到 -
... ...
#gzip on;
server {
listen 80;
server_name localhost;
#charset koi8-r;
... ...
把上面的 80 修改为你想要的端口，如：8080 。
修改配置后验证配置是否合法：
[root@rhel0 nginx-1.22.0]# /usr/local/nginx/sbin/nginx -t
nginx: the configuration file /usr/local/nginx/conf/nginx.conf syntax is ok
nginx: configuration file /usr/local/nginx/conf/nginx.conf test is successful
启动Nginx程序、查看进程 -
[root@rhel0 nginx-1.22.0]# /usr/local/nginx/sbin/nginx
[root@rhel0 nginx-1.22.0]# ps -ef | grep nginx
root 21348 24564 0 06:40 pts/0 00:00:00 grep --color=auto nginx
root 30901 1 0 Jan05 ? 00:00:00 nginx: master process /usr/local/nginx/sbin/nginx
nginx 30902 30901 0 Jan05 ? 00:00:00 nginx: worker process
nginx停止、重启
未添加nginx服务前对nginx的管理只能通过一下方式管理：
# nginx 管理的几种方式 -
# 启动Nginx
/usr/local/nginx/sbin/nginx
# 从容停止Nginx：
kill -QUIT 主进程号 # 如上一步中的 ps 命令输出的 29151，就是 Nginx的主进程号
# 快速停止Nginx：
kill -TERM 主进程号
# 强制停止Nginx：
pkill -9 nginx
# 平滑重启nginx
/usr/nginx/sbin/nginx -s reload
#设置NGINX开机自动启动
echo "/usr/local/nginx/sbin/nginx" >> /etc/rc.local
现在我们来看看安装的Nginx的运行结果，可以简单地使用curl命令访问rhel0测试，结果如下 -
[root@rhel0 nginx-1.22.0]# curl rhel0
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
body {
width: 35em;
margin: 0 auto;
font-family: Tahoma, Verdana, Arial, sans-serif;
}
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>
<p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p>
<p><em>Thank you for using nginx.</em></p>
</body>
</html>
2. keepalivad安装
2.1 安装前工作
依赖包安装
[root@rhel0 ~]# yum -y install libnl libnl-devel libnfnetlink-devel e2fsprogs-devel keyutils-libs-devel libsepol-devel libselinux-devel krb5-devel zlib-devel openssl-devel
如果用介质挂载yum源，libnfnetlink-devel可能会提示不存在，可通过下方链接寻找对应版本的rpm包，然后直接rpm安装即可
Libnfnetlink-devel Download (EOPKG, RPM, XBPS) (pkgs.org)
[root@rhel0 ~]# rpm -ivh libnfnetlink-devel-1.0.1-4.el7.x86_64.rpm
warning: libnfnetlink-devel-1.0.1-4.el7.x86_64.rpm: Header V3 RSA/SHA256 Signature, key ID f4a80eb5: NOKEY
Preparing... ################################# [100%]
Updating / installing...
1:libnfnetlink-devel-1.0.1-4.el7 ################################# [100%]
修改内核参数
[root@rhel0 ~]# vim /etc/sysctl.conf
# sysctl settings are defined through files in
# /usr/lib/sysctl.d/, /run/sysctl.d/, and /etc/sysctl.d/.
#
# Vendors settings live in /usr/lib/sysctl.d/.
# To override a whole file, create a new file with the same in
# /etc/sysctl.d/ and put new settings there. To override
# only specific settings, add a file with a lexically later
# name in /etc/sysctl.d/ and put new settings there.
#
# For more information, see sysctl.conf(5) and sysctl.d(5).
net.ipv4.tcp_max_syn_backlog=8192
net.ipv4.tcp_tw_reuse=1
net.ipv4.tcp_max_syn_backlog=8192
net.ipv4.tcp_keepalive_time=1800
net.ipv4.tcp_fin_timeout=30
net.core.rmem_max=16777216
net.core.wmem_max=16777216
net.ipv4.tcp_rmem=4096 87380 16777216
net.ipv4.tcp_wmem=4096 65536 16777216
# 禁用ARP，增大backlog并发数
net.ipv4.conf.all.arp_ignore=1
net.ipv4.conf.all.arp_announce=2
net.core.netdev_max_backlog=500000
[root@rhel0 ~]# sysctl -p
2.2 下载keepalived安装源文件
源码下载，可官网下载地址：Keepalived for Linux下载并上传到服务器(这里选择最新稳定版本：version 2.2.7)，如下图所示
或直接在服务上执行以下命令下载
[root@rhel0 ~]# cd /usr/local/src
[root@rhel0 src]# wget -c https://www.keepalived.org/software/keepalived-2.2.7.tar.gz
解压上面下载的文件
[root@rhel0 src]# tar -xzvf keepalived-2.2.7.tar.gz
2.3 安装服务
[root@rhel0 src]# cd keepalived-2.2.7/
[root@rhel0 keepalived-2.2.7]# ./configure --prefix=/usr/local/keepalived
编译结果见下图，Use IPVS Framework、Use VRRP Framework必须为YES
[root@rhel0 keepalived-2.2.7]# make
[root@rhel0 keepalived-2.2.7]# make install
[root@rhel1 keepalived-2.2.7]# cp /usr/local/src/keepalived-2.2.7/keepalived/etc/init.d/keepalived /etc/rc.d/init.d/
[root@rhel0 keepalived-2.2.7]# chmod +x /etc/init.d/keepalived
[root@rhel1 init.d]# vim /etc/init.d/keepalived
#修改/etc/init.d/keepalived, 寻找大约15行左右的. /etc/sysconfig/keepalived, 修改为：
#. /usr/local/keepalived/etc/sysconfig/keepalived, 即指向正确的安装文件位置
[root@rhel0 keepalived-2.2.7]# vim ~/.bash_profile
#将keepavlied主程序所在路径导入到环境变量PATH中
# User specific environment and startup programs
PATH=$PATH:$HOME/bin:/usr/local/keepalived/sbin
export PATH
[root@rhel0 keepalived-2.2.7]# source ~/.bash_profile
#修改/usr/local/keepalived/etc/sysconfig/keepalived文件，设置正确的服务启动参数
[root@rhel0 keepalived-2.2.7]# vim /usr/local/keepalived/etc/sysconfig/keepalived
KEEPALIVED_OPTIONS="-D -f /usr/local/keepalived/etc/keepalived/keepalived.conf"
#设置keeplived开机自动启动
[root@rhel0 keepalived-2.2.7]# systemctl enable keepalived.service
2.4 主Keepalived配置
[root@rhel0 ~]# vim /usr/local/keepalived/etc/keepalived/keepalived.conf
! Configuration File for keepalived #全局定义
global_defs {
notification_email { #指定keepalived在发生事件时(比如切换)发送通知邮件的邮箱
111@qq.com #设置报警邮件地址，可以设置多个，每行一个。 需开启本机的sendmail服务
}
notification_email_from xiaochong@then.com #keepalived在发生诸如切换操作时需要发送email通知地址
smtp_server 127.0.0.1 #指定发送email的smtp服务器
smtp_connect_timeout 30 #设置连接smtp server的超时时间
router_id HAmaster-130 #运行keepalived的机器的一个标识，通常可设为hostname。故障发生时，发邮件时显示在邮件主题中的信息。
}
vrrp_script chk_http_port { #检测nginx服务是否在运行。有很多方式，比如进程，用脚本检测等等
script "/usr/local/keepalived/ch_nginx.sh" #这里通过脚本监测
interval 2 #脚本执行间隔，每2s检测一次
weight -5 #脚本结果导致的优先级变更，检测失败（脚本返回非0）则优先级 -5
fall 2 #检测连续2次失败才算确定是真失败。会用weight减少优先级（1-255之间）
rise 1 #检测1次成功就算成功。但不修改优先级
}
vrrp_instance VI_1 { #keepalived在同一virtual_router_id中priority（0-255）最大的会成为master，也就是接管VIP，当priority最大的主机发生故障后次priority将会接管
state MASTER #指定keepalived的角色，MASTER表示此主机是主服务器，BACKUP表示此主机是备用服务器。注意这里的state指定instance(Initial)的初始状态，就是说在配置好后，这台服务器的初始状态就是这里指定的，但这里指定的不算，还是得要通过竞选通过优先级来确定。如果这里设置为MASTER，但如若他的优先级不及另外一台，那么这台在发送通告时，会发送自己的优先级，另外一台发现优先级不如自己的高，那么他会就回抢占为MASTER
interface ens33 #指定HA监测网络的接口。实例绑定的网卡，因为在配置虚拟IP的时候必须是在已有的网卡上添加的
mcast_src_ip 192.168.111.157 # 发送多播数据包时的源IP地址，这里注意了，这里实际上就是在哪个地址上发送VRRP通告，这个非常重要，一定要选择稳定的网卡端口来发送，这里相当于heartbeat的心跳端口，如果没有设置那么
就用默认的绑定的网卡的IP，也就是interface指定的IP地址
virtual_router_id 51 #虚拟路由标识，这个标识是一个数字，同一个vrrp实例使用唯一的标识。即同一vrrp_instance下，MASTER和BACKUP必须是一致的
priority 101 #定义优先级，数字越大，优先级越高，在同一个vrrp_instance下，MASTER的优先级必须大于BACKUP的优先级
advert_int 1 #设定MASTER与BACKUP负载均衡器之间同步检查的时间间隔，单位是秒
authentication { #设置验证类型和密码。主从必须一样
auth_type PASS #设置vrrp验证类型，主要有PASS和AH两种
auth_pass 1111 #设置vrrp验证密码，在同一个vrrp_instance下，MASTER与BACKUP必须使用相同的密码才能正常通信
}
virtual_ipaddress { #VRRP HA 虚拟地址 如果有多个VIP，继续换行填写
192.168.111.158
}
track_script { #执行监控的服务。注意这个设置不能紧挨着写在vrrp_script配置块的后面（实验中碰过的坑），否则nginx监控失效！！
chk_http_port #引用VRRP脚本，即在 vrrp_script 部分指定的名字。定期运行它们来改变优先级，并最终引发主备切换。
}
}
#配置检测脚本
[root@rhel0 ~]# vim /usr/local/keepalived/ch_nginx.sh
#!/bin/bash
counter=$(ps -C nginx --no-heading|wc -l)
echo "$counter"
if [ "${counter}" = "0" ]; then
/usr/local/nginx/sbin/nginx -c /usr/local/nginx/conf/nginx.conf
sleep 2
counter=$(ps -C nginx --no-heading|wc -l)
if [ "${counter}" = "0" ]; then
/etc/init.d/keepalived stop
fi
fi
[root@rhel0 ~]# chmod 755 /usr/local/keepalived/ch_nginx.sh
2.5 备Keepalived配置
[root@rhel1 ~]# vim /usr/local/keepalived/etc/keepalived/keepalived.conf
! Configuration File for keepalived
global_defs {
notification_email {
111@qq.com
}
notification_email_from xiaochong@then.com
smtp_server 127.0.0.1
smtp_connect_timeout 30
router_id HAbackup-129
}
vrrp_script chk_http_port {
script "/usr/local/keepalived/ch_nginx.sh"
interval 2
weight -5
fall 2
rise 1
}
vrrp_instance VI_1 {
state BACKUP
interface ens33
mcast_src_ip 192.168.111.156
virtual_router_id 51
priority 99
advert_int 1
authentication {
auth_type PASS
auth_pass 1111
}
virtual_ipaddress {
192.168.111.158
}
track_script {
chk_http_port
}
}
#配置检测脚本
[root@rhel1 ~]# vim /usr/local/keepalived/ch_nginx.sh
#!/bin/bash
counter=$(ps -C nginx --no-heading|wc -l)
echo "$counter"
if [ "${counter}" = "0" ]; then
/usr/local/nginx/sbin/nginx -c /usr/local/nginx/conf/nginx.conf
sleep 2
counter=$(ps -C nginx --no-heading|wc -l)
if [ "${counter}" = "0" ]; then
/etc/init.d/keepalived stop
fi
fi
[root@rhel1 ~]# chmod 755 /usr/local/keepalived/ch_nginx.sh
3.keepalivad 功能及灾备验证
3.1 功能性验证
访问vip地址，查看是否能联通nginx服务
3.2 灾备验证
1、先后在master、slave服务器上启动nginx和keepalived，保证这两个服务都正常开启
[root@rhel0 ~]# systemctl start keepalived.service
[root@rhel0 ~]# systemctl status keepalived.service
[root@rhel0 ~]# /usr/local/nginx/sbin/nginx
[root@rhel1 ~]# systemctl start keepalived.service
[root@rhel1 ~]# systemctl status keepalived.service
[root@rhel0 ~]# /usr/local/nginx/sbin/nginx
2、在主服务器上查看是否已经绑定了虚拟IP
先查看master
再看backup
3、停止主服务器上的keepalived
[root@rhel0 ~]# systemctl stop keepalived.service
[root@rhel0 ~]# systemctl status keepalived.service
● keepalived.service - LVS and VRRP High Availability Monitor
Loaded: loaded (/usr/lib/systemd/system/keepalived.service; enabled; vendor preset: disabled)
Active: inactive (dead) since Fri 2022-05-27 15:55:06 CST; 1min 5s ago
Docs: man:keepalived(8)
man:keepalived.conf(5)
man:genhash(1)
https://keepalived.org
Main PID: 7687 (code=exited, status=0/SUCCESS)
Tasks: 2
CGroup: /system.slice/keepalived.service
├─7694 nginx: master process /usr/local/nginx/sbin/nginx -c /usr/local/nginx/conf/nginx.conf
└─7696 nginx: worker process
May 27 15:38:33 rhel0 Keepalived_vrrp[7688]: Sending gratuitous ARP on ens33 for 192.168.111.158
May 27 15:38:33 rhel0 Keepalived_vrrp[7688]: Sending gratuitous ARP on ens33 for 192.168.111.158
May 27 15:55:05 rhel0 Keepalived[7687]: Stopping
May 27 15:55:05 rhel0 systemd[1]: Stopping LVS and VRRP High Availability Monitor...
May 27 15:55:05 rhel0 Keepalived_vrrp[7688]: (VI_1) sent 0 priority
May 27 15:55:05 rhel0 Keepalived_vrrp[7688]: (VI_1) removing VIPs.
May 27 15:55:06 rhel0 Keepalived_vrrp[7688]: Stopped - used (self/children) 0.017705/3.373097 user time, 0.365595/3.912586 system time
May 27 15:55:06 rhel0 Keepalived[7687]: CPU usage (self/children) user: 0.000000/3.390802 system: 0.000918/4.278978
May 27 15:55:06 rhel0 Keepalived[7687]: Stopped Keepalived v2.2.7 (01/16,2022)
May 27 15:55:06 rhel0 systemd[1]: Stopped LVS and VRRP High Availability Monitor.
4、查看backup服务是否接管了vip地址，可以看到backup服务器已经接管了vip地址
此时访问vip地址可以看到服务还是正常可以访问的
5、重新启动主服务器上的keepalived，发现主服务器又重新接管了VIP，此时slave机器上的VIP已经不在了
[root@rhel0 ~]# systemctl start keepalived.service
[root@rhel0 ~]# ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: ens33: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
link/ether 00:0c:29:8d:af:16 brd ff:ff:ff:ff:ff:ff
inet 192.168.111.157/24 brd 192.168.111.255 scope global noprefixroute dynamic ens33
valid_lft 1251sec preferred_lft 1251sec
inet 192.168.111.158/32 scope global ens33
valid_lft forever preferred_lft forever
inet6 fe80::93db:1d0e:e71e:3da4/64 scope link noprefixroute
valid_lft forever preferred_lft forever
3: virbr0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default qlen 1000
link/ether 52:54:00:77:0f:01 brd ff:ff:ff:ff:ff:ff
inet 192.168.122.1/24 brd 192.168.122.255 scope global virbr0
valid_lft forever preferred_lft forever
4: virbr0-nic: <BROADCAST,MULTICAST> mtu 1500 qdisc pfifo_fast master virbr0 state DOWN group default qlen 1000
link/ether 52:54:00:77:0f:01 brd ff:ff:ff:ff:ff:ff
版权声明：本文为yangZHyu原创文章，遵循CC 4.0 BY-SA版权协议，转载请附上原文出处链接和本声明。