Prometheus+Grafana

基本说明

Prometheus+Grafana无疑是目前最流行的监控预警系统，这个系统是基于GO语言运行的，所有我们首先需要配置Go语言的环境，这里涉及到我另一篇文章的内容，我就先不作重述了。
链接地址为：
https://blog.csdn.net/weixin_48803304/article/details/107296099

1.GO环境配置

这里提供一个我整合以后的脚本文件

#!/bin/bash
PATH=/bin:/sbin:/usr/bin:/usr/sbin:/usr/local/bin:/usr/local/sbin:~/bin
export PATH
cur_dir=$(pwd)

# 定义显示文本色彩
Color_Text()
{
    echo -e " \e[0;$2m$1\e[0m"
}

Echo_Red()
{
    echo $(Color_Text "$1" "31")
}

Echo_Green()
{
    echo $(Color_Text "$1" "32")
}

Echo_Yellow()
{
    echo $(Color_Text "$1" "33")
}

Echo_Blue()
{
    echo $(Color_Text "$1" "34")
}
Echo_Blue '[info] 安装Golang'
if [ ! -d "/tools" ];then
	mkdir /tools
fi
cd /tools
if [ ! -f 'go1.17.linux-amd64.tar.gz' ];then
	Echo_Blue '[info] 开始下载安装包'
	wget https://dl.google.com/go/go1.17.linux-amd64.tar.gz
else
	Echo_Yellow "[info] 安装包已经存在，继续执行..."
fi
if [ ! -d "/usr/local/go" ];then
	Echo_Blue '[info] 解压安装包'
	tar -C /usr/local -xzf go1.17.linux-amd64.tar.gz
else
	Echo_Yellow "[info] 文件夹/usr/local/go已经存在，继续执行..."
fi
if [ ! -f "/etc/profile.d/golang.sh" ];then
    Echo_Blue '[info] 写入变量'
cat >>/etc/profile.d/golang.sh<<EOF
export GOROOT=/usr/local/go
export PATH=\$PATH:\$GOROOT/bin
export GOPATH=/root/go
export PATH=\$PATH:\$GOPATH/BIN
EOF
else
    Echo_Yellow "[info] GO环境变量已经存在，继续执行..."
fi
source /etc/profile.d/golang.sh
source /etc/profile.d/golang.sh

可以执行go version查看结果

[root@localhost tools]# go version
go version go1.15.5 linux/amd64

2.配置Prometheus

官网为https://prometheus.io/，我们进入下载页面，下载我们需要的包或是扩展https://prometheus.io/download/

mkdir /home/prometheus

为了方便管理和维护，我把所有的prometheus项目及控件放到一个目录中，首先我们下载prometheus

cd /home/prometheus/
wget https://github.com/prometheus/prometheus/releases/download/v2.28.1/prometheus-2.28.1.linux-amd64.tar.gz
mkdir prometheus && tar zxvf prometheus-2.28.1.linux-amd64.tar.gz -C ./prometheus --strip-components 1
cd prometheus/

目录中就是我们需要的文件，我们可以将prometheus设置为一项服务

cat >>/etc/systemd/system/prometheus.service<<EOF
[Unit]
Description=https://prometheus.io/

[Service]
Type=simple
MemoryLimit=100M
CPUQuota=20%

WorkingDirectory=/home/prometheus/prometheus/
ExecStart=/home/prometheus/prometheus/prometheus --config.file=/home/prometheus/prometheus/prometheus.yml
Restart=on-failure
[Install]
WantedBy=multi-user.target
EOF

添加防火墙规则并启动服务

firewall-cmd --permanent --zone=public --add-port=9090/tcp
firewall-cmd --reload
systemctl enable prometheus
systemctl start prometheus
systemctl status prometheus

这个时候我们就可以通过http方式访问我们的服务了
不同的扩展需要到不同的地方下载，官方提供了入口
https://prometheus.io/docs/instrumenting/exporters/#databases

Prometheus进阶

1.node_exporter

在Prometheus的架构设计中，Prometheus Server并不直接服务监控特定的目标，其主要任务负责数据的收集，存储并且对外提供数据查询支持。因此为了能够能够监控到某些东西，如主机的CPU使用率，我们需要使用到Exporter。Prometheus周期性的从exporter暴露的HTTP服务地址（通常是/metrics）拉取监控样本数据。

exporter可以是一个相对开放的概念，本质是一个独立运行的程序独立于监控目标以外，也可以是直接内置在监控目标中。只要能够向Prometheus提供标准格式的监控样本数据(TS时间序列)即可。
为了能够采集到主机的运行指标如CPU, 内存，磁盘等信息。就需要用到node_exporter。
下载并解压缩包

cd /home/prometheus
wget https://github.com/prometheus/node_exporter/releases/download/v1.1.2/node_exporter-1.1.2.linux-amd64.tar.gz
tar zxvf node_exporter-1.1.2.linux-amd64.tar.gz
mv node_exporter-1.1.2.linux-amd64 node_exporter
cd node_exporter/

设置系统服务

cat >>/etc/systemd/system/node_exporter.service<<EOF
[Unit]
Description=node_exporter
After=network.target 

[Service]
Type=simple
MemoryLimit=100M
CPUQuota=20%
ExecStart=/home/prometheus/node_exporter/node_exporter

[Install]
WantedBy=multi-user.target


EOF

添加防火墙规则并启动服务

firewall-cmd --permanent --zone=public --add-port=9100/tcp
firewall-cmd --reload
systemctl enable node_exporter
systemctl start node_exporter
systemctl status node_exporter

常用的模板ID8919

2.安装grafana

grafana官网地址为https://grafana.com/
进入下载页面，可以找到不同的系统的安装方法，我直接使用 centos的安装方法
https://grafana.com/grafana/download?pg=get&plcmt=selfmanaged-box1-cta1

wget https://dl.grafana.com/oss/release/grafana-8.0.5-1.x86_64.rpm
sudo yum install -y grafana-8.0.5-1.x86_64.rpm
grafana-cli plugins install grafana-image-renderer
yum -y install libatk-bridge* libXss* libgtk*
firewall-cmd --permanent --zone=public --add-port=3000/tcp
firewall-cmd --reload
systemctl enable grafana-server
systemctl start grafana-server
systemctl status grafana-server

关于grafana绑定prometheus的例子网上有很多，感兴趣的朋友可以搜索一下

SMTP设置

vi /etc/grafana/grafana.ini

#################################### SMTP / Emailing ##########################
[smtp]
enabled = true
host = smtp.qq.com:465
user = lzyats@qq.com
# If the password contains # or ; you have to wrap it with triple quotes. Ex """#password;"""
password = 你的邮箱授权码
;cert_file =
;key_file =
skip_verify = true
from_address = lzyats@qq.com
from_name = Grafana
# EHLO identity in SMTP dialog (defaults to instance_name)
;ehlo_identity = dashboard.example.com
# SMTP startTLS policy (defaults to 'OpportunisticStartTLS')
#startTLS_policy = NoStartTLS

[emails]
;welcome_email_on_sign_up = false
;templates_pattern = emails/*.html

如果想在钉钉报警中带图片，记得配置图片地址

[external_image_storage]
provider = local

3.grafana模板

由于grafana是个开放的平台，自然会有许多模板提供，地址是
https://grafana.com/grafana/dashboards
我们关注一下node_exporter的模板，推荐使用9276

mysql_exporter.service

vi /etc/systemd/system/mysqld_exporter81.service

[Unit]
Description=mysqld_exporter81
After=network.target
[Service]
Type=simple
User=mysql
# exporter对应授权账号,exporter123对应授权密码,localhost对应授权账号密码所在的地址
Environment=DATA_SOURCE_NAME=exporter:123456@(l192.168.200.130:3306)/
ExecStart=/usr/local/mysqld_exporter/mysqld_exporter --web.listen-address=0.0.0.0:9104
  --config.my-cnf /home/data/mysql80/my1.cnf \
  --collect.slave_status \
  --collect.slave_hosts \
  --log.level=error \
  --collect.info_schema.processlist \
  --collect.info_schema.innodb_metrics \
  --collect.info_schema.innodb_tablespaces \
  --collect.info_schema.innodb_cmp \
  --collect.info_schema.innodb_cmpmem
Restart=on-failure
[Install]
WantedBy=multi-user.targe

systemctl enable mysqld_exporter81
systemctl start mysqld_exporter81
systemctl status mysqld_exporter81

vi /etc/systemd/system/mysqld_exporter82.service

[Unit]
Description=mysqld_exporter82
After=network.target
[Service]
Type=simple
User=mysql
# exporter对应授权账号,exporter123对应授权密码,localhost对应授权账号密码所在的地址
Environment=DATA_SOURCE_NAME=exporter:123456@(l192.168.200.130:3308)/
ExecStart=/usr/local/mysqld_exporter/mysqld_exporter --web.listen-address=0.0.0.0:9105
  --config.my-cnf /home/data/mysql80/my2.cnf \
  --collect.slave_status \
  --collect.slave_hosts \
  --log.level=error \
  --collect.info_schema.processlist \
  --collect.info_schema.innodb_metrics \
  --collect.info_schema.innodb_tablespaces \
  --collect.info_schema.innodb_cmp \
  --collect.info_schema.innodb_cmpmem
Restart=on-failure
[Install]
WantedBy=multi-user.targe

systemctl enable mysqld_exporter82
systemctl start mysqld_exporter82
systemctl status mysqld_exporter82

alertmanager

vi /etc/systemd/system/alertmanager.service

[Unit]
Description=alertmanager
Documentation=https://prometheus.io/
After=network.target

[Service]
Type=simple
WorkingDirectory=/home/prometheus/alertmanager/
ExecStart=/home/prometheus/alertmanager/alertmanager --config.file=/home/prometheus/alertmanager/alertmanager.yml
#Restart=always
RestartSec=60
Restart=on-failure

[Install]
WantedBy=multi-user.target

systemctl enable alertmanager
systemctl start alertmanager
systemctl status alertmanager

apache_exporter

vi /etc/systemd/system/apache_exporter.service

[Unit]
Description=Prometheus apache exporter
Documentation=https://github.com/Lusitaniae/apache_exporter
After=network.target

[Service]
Type=simple
ExecStart=/usr/local/apache_exporter/apache_exporter --scrape_uri=http://lzyats:lijiangkkk@127.0.0.1:8081/server-status?auto \
--telemetry.address=0.0.0.0:9117 \
--telemetry.endpoint=/metrics
Restart=on-failure

[Install]
WantedBy=multi-user.target

systemctl enable apache_exporter
systemctl start apache_exporter
systemctl status apache_exporter

redis_exporter

这个需要单独下载，下载地址为：https://github.com/oliver006/redis_exporter/releases/download/v1.24.0/redis_exporter-v1.24.0.linux-amd64.tar.gz
vi /etc/systemd/system/redis_exporter.service

[Unit]
Description=redis_exporter
Documentation=https://github.com/oliver006/redis_exporter
After=network.target
 
[Service]
Type=simple
MemoryLimit=100M
CPUQuota=20%
WorkingDirectory=/home/prometheus/redis_exporter
ExecStart=/home/prometheus/redis_exporter/redis_exporter -redis.addr=127.0.0.1:6379 -redis.password=lijiangkkk

Restart=on-failure

[Install]
WantedBy=multi-user.target

firewall-cmd --permanent --zone=public --add-port=9121/tcp
firewall-cmd --reload
systemctl enable redis_exporter
systemctl start redis_exporter
systemctl status redis_exporter

有一点需要注意的是1.0版本以上，多实例的配置方法改变为

scrape_configs:
  - job_name: 'redis_exporter_targets'
    static_configs:
      - targets:
        - redis://192.168.200.130:6379
        - redis://192.168.200.130:6389
        - redis://192.168.200.131:6379
        - redis://192.168.200.131:6389
    metrics_path: /scrape
    relabel_configs:
      - source_labels: [__address__]
        target_label: __param_target
      - source_labels: [__param_target]
        target_label: instance
      - target_label: __address__
        replacement: 127.0.0.1:9121

  - job_name: 'redis_exporter'
    static_configs:
      - targets:
        - 127.0.0.1:9121

nginx-vts-exporter

1.安装nginx 插件

我使用的是tengine2.33，重新编译一下，加上需要的扩展

cd /tools/
git clone git://github.com/vozlt/nginx-module-vts.git
cd tengine-2.3.3
	./configure --prefix=/etc/nginx --with-http_ssl_module --with-http_realip_module --with-http_stub_status_module --user=nginx --group=nginx --add-module=/tools/nginx-module-vts
	make && make install
	mkdir /var/log/nginx
	mkdir /etc/nginx/conf/conf.d

接下来配置nginx

http {
    vhost_traffic_status_zone;
    vhost_traffic_status_filter_by_host on;
    server {
    #   vhost_traffic_status off;
        location /status {
            vhost_traffic_status_display;
            vhost_traffic_status_display_format html;
        }
    }
}

docker运行方法

docker run -d -p 9913:9913 --env NGINX_STATUS="http://192.168.201.12/status/format/json" --name gh-nginx sophos/nginx-vts-exporter

2.安装nginx-vts-exporter

nginx-vts-exporter扩展需要进入官网的其他区域查找，进入https://prometheus.io/docs/instrumenting/exporters/找到以下部分
在这里插入图片描述
点击下载最新的版本，最好是下载编译好的包

wget https://github.com/hnlq715/nginx-vts-exporter/releases/download/v0.10.3/nginx-vts-exporter-0.10.3.linux-amd64.tar.gz
tar zxvf nginx-vts-exporter-0.10.3.linux-amd64.tar.gz
mv nginx-vts-exporter-0.10.3.linux-amd64 /home/prometheus/nginx-vts-exporter

设置为启动服务

vi /etc/systemd/system/nginx-vts-exporter.service

[Unit]
Description=nginx-vts-exporter
Documentation=https://github.com/prometheus/redis_exporter
After=network.target
 
[Service]
Type=simple
MemoryLimit=100M
CPUQuota=20%
WorkingDirectory=/home/prometheus/nginx-vts-exporter
ExecStart=/home/prometheus/nginx-vts-exporter/nginx-vts-exporter -nginx.scrape_uri=http://127.0.0.1/status/format/json

Restart=on-failure

[Install]
WantedBy=multi-user.target

firewall-cmd --permanent --zone=public --add-port=9913/tcp
firewall-cmd --reload
systemctl enable nginx-vts-exporter
systemctl start nginx-vts-exporter
systemctl status nginx-vts-exporter

导入模板2949

gearman-exporter

wget https://github.com/bakins/gearman-exporter/releases/download/v0.5.0/gearman-exporter.linux.amd64
mv gearman-exporter.linux.amd64 gearman-exporter
chmod +x gearman-exporter

vi /etc/systemd/system/gearman-exporter.service

[Unit]
Description=gearman-exporter
Documentation=https://github.com/bakins/gearman-exporter
After=network.target
 
[Service]
Type=simple
MemoryLimit=100M
CPUQuota=20%
WorkingDirectory=/home/prometheus/
ExecStart=/home/prometheus/gearman-exporter --addr 192.168.200.131:9418

Restart=on-failure

[Install]
WantedBy=multi-user.target

firewall-cmd --permanent --zone=public --add-port=9418/tcp
firewall-cmd --reload
systemctl enable gearman-exporter
systemctl start gearman-exporter
systemctl status gearman-exporter

模板11423

版权声明：本文为weixin_48803304原创文章，遵循CC 4.0 BY-SA版权协议，转载请附上原文出处链接和本声明。