Prometheus监控安装

2019-12-12 2026-09-04 约 2015 字预计阅读 5 分钟次阅读条评论

Prometheus, a Cloud Native Computing Foundation project, is a systems and service monitoring system. It collects metrics from configured targets at given intervals, evaluates rule expressions, displays the results, and can trigger alerts when specified conditions are observed.

The features that distinguish Prometheus from other metrics and monitoring systems are:

A multi-dimensional data model (time series defined by metric name and set of key/value dimensions)
PromQL, a powerful and flexible query language to leverage this dimensionality
No dependency on distributed storage; single server nodes are autonomous
An HTTP pull model for time series collection
Pushing time series is supported via an intermediary gateway for batch jobs
Targets are discovered via service discovery or static configuration
Multiple modes of graphing and dashboarding support
Support for hierarchical and horizontal federation

1 配置Prometheus

1.1 下载

wget https://github.com/prometheus/prometheus/releases/download/v3.5.0/prometheus-3.5.0.linux-amd64.tar.gz
tar zxf prometheus-3.5.0.linux-amd64.tar.gz
mv prometheus-3.5.0.linux-amd64 /usr/local/prometheus

1.2 创建用户

groupadd --system prometheus
useradd --system -g prometheus -s /sbin/nologin -c "Prometheus Monitoring System" prometheus

1.3 赋权

chown -R prometheus:prometheus /usr/local/prometheus

1.4 创建数据目录

mkdir /data/prometheus
chown -R prometheus:prometheus /data/prometheus

1.5 创建服务

cat > /usr/lib/systemd/system/prometheus.service <<EOF
[Unit]
Description=Prometheus
After=network.target
[Service]
Type=simple
User=prometheus
ExecStart=/usr/local/prometheus/prometheus \
--config.file=/usr/local/prometheus/prometheus.yml \
--storage.tsdb.path=/data/prometheus \
--storage.tsdb.retention.time=30d \	
--storage.tsdb.retention.size=512M \
--web.enable-admin-api \
--web.enable-lifecycle \
--web.external-url=http://monitor.example.com
ExecReload=/bin/kill -HUP $MAINPID
Restart=on-failure
[Install]
WantedBy=multi-user.target
EOF

Type设置为notify时，服务会不断重启

--storage.tsdb.path是可选项，默认数据目录在运行目录的./dada目录中

--storage.tsdb.retention.time设置保留多长时间的数据

--storage.tsdb.retention.size存储块可以使用的最大字节数（请注意，这不包括WAL大小，这可能很大）。最早的数据将被删除。默认为0或禁用。此标志是实验性的，可以在将来的版本中进行更改。支持的单位：KB，MB，GB，PB。例如：“512MB”

--web.enable-admin-api开启对admin api的访问权限

--web.enable-lifecycle启用远程热加载配置文件

--web.external-url=http://localhost:9090/ prometheus主机外网地址，不写会导致告警GeneratorURL不对

1.6 修改Prometheus配置

vi prometheus.yml

# Alertmanager configuration
alerting:
  alertmanagers:
  - static_configs:
    - targets:
       - 127.0.0.1:9093

# Load rules once and periodically evaluate them according to the global 'evaluation_interval'.
rule_files:
  - "rules/*.yml"

1.7 启动

systemctl enable prometheus.service
systemctl start prometheus.service

1.8 prometheus自动发现

自动发现机制方便我们在监控系统中动态的添加或者删除资源。比如zabbix可以自动发现监控主机以及监控资源。prometheus作为一个可以与zabbix旗鼓相当的监控系统，自然也有它的自动发现机制。

1.8.1 file_sd_configs

file_sd_configs可以用来动态的添加和删除target。

修改prometheus的配置文件

  - job_name: 'node'
    file_sd_configs:
    - refresh_interval: 1m
      files:
      - targets/nodes/*.yml

创建被扫描的文件nodes.yml

- targets:
  - '172.19.179.253:9100'
  - '172.19.179.254:9100'
  labels:
    server: linux

1.8.2 consul_sd_file

Consul 是基于 GO 语言开发的开源工具，主要面向分布式，服务化的系统提供服务注册、服务发现和配置管理的功能。Consul 提供服务注册/发现、健康检查、Key/Value存储、多数据中心和分布式一致性保证等功能。之前我们通过 Prometheus 实现监控，当新增一个 Target 时，需要变更服务器上的配置文件，即使使用 file_sd_configs 配置，也需要登录服务器修改对应 Json 文件，会非常麻烦。不过 Prometheus 官方支持多种自动服务发现的类型，其中就支持 Consul。

consul的配置需要有consul的服务提供

修改prometheus的配置文件

  - job_name: 'consul-prometheus'
    consul_sd_configs:
    - server: '172.30.12.167:8500'
      services: []

2 配置node_exporter监控主机

Node_exporter是可以在* Nix和Linux系统上运行的计算机度量标准的导出器。

Node_exporter 主要用于暴露 metrics 给 Prometheus，其中 metrics 包括：cpu 的负载，内存的使用情况，网络等。

2.1 下载

wget https://github.com/prometheus/node_exporter/releases/download/v0.18.1/node_exporter-0.18.1.linux-amd64.tar.gz

tar zxf node_exporter-0.18.1.linux-amd64.tar.gz
mv node_exporter-0.18.1.linux-amd64 /usr/local/node_exporter

2.2 创建node_exporter服务

cat > /usr/lib/systemd/system/node_exporter.service <<EOF
[Unit]
Description=Node Exporter
After=network.target

[Service]
ExecStart=/usr/local/node_exporter/node_exporter

[Install]
WantedBy=multi-user.target
EOF

2.3 启动

systemctl enable node_exporter.service
systemctl start node_exporter.service

2.4 容器运行

docker run -d \
  --net=host \
  -v "/:/host:ro,rslave" \
  prom/node-exporter:latest \
  --path.rootfs=/host \
  --collector.filesystem.ignored-mount-points="^/(sys|proc|dev|host|etc)($|/)"

3 Blackbox_exporter

3.1 下载

wget https://github.com/prometheus/blackbox_exporter/releases/download/v0.17.0/blackbox_exporter-0.17.0.linux-amd64.tar.gz

tar zxf blackbox_exporter-0.17.0.linux-amd64.tar.gz
mv blackbox_exporter-0.17.0.linux-amd64 blackbox_exporter

修改Prometheus配置，增加blackbox-http，使用基于文件的自动发现

  - job_name: "blackbox-http"
    metrics_path: /probe
    params:
      module: [http_2xx]
    file_sd_configs:
    - refresh_interval: 1m
      files:
      - targets/blackbox/http.yml
    relabel_configs:
    - source_labels: [__address__]
      target_label: __param_target
    - source_labels: [__param_target]
      target_label: instance
    - target_label: __address__
      replacement: 127.0.0.1:9115  # blackbox服务地址

创建targets/blackbox/http.yml文件

- targets:
  - https://baidu.com

3.2 创建服务

cat > /usr/lib/systemd/system/blackbox.service <<EOF
[Unit]
Description=blackbox_exporter
After=network.target

[Service]
User=root
Type=simple
ExecStart=/usr/local/blackbox_exporter/blackbox_exporter --config.file=/usr/local/blackbox_exporter/blackbox.yml
Restart=on-failure

[Install]
WantedBy=multi-user.target
EOF

3.3 启动服务

systemctl start blackbox.service
systemctl enable blackbox.service

3.4 重载Prometheus

curl -X POST "http://127.0.0.1:9090/-/reload"

3.5 grafana图表

导入 https://grafana.com/grafana/dashboards/9965

3.6 告警配置

创建rules/blackbox_exporter.yml文件

4 配置Alertmanager

4.1 下载

wget https://github.com/prometheus/alertmanager/releases/download/v0.20.0/alertmanager-0.20.0.linux-amd64.tar.gz

tar zxf alertmanager-0.20.0.linux-amd64.tar.gz

4.2 修改Alertmanager配置

vi alertmanager.yml

global:
  resolve_timeout: 5m
# 路由树: 根节点
route:
  receiver: webhook
  # 分组维度
  group_by: [alertname]
  # 新分组等待发送, 收敛间隔30s
  group_wait: 30s
  # 存在分组,有新告警加入发送, 收敛间隔5m
  group_interval: 5m
  # 发送成功的alert重复发送需等待3h
  repeat_interval: 3h
  routes:
  - receiver: webhook
    group_wait: 10s
# 接收
receivers:
- name: webhook
  webhook_configs:
  - url: http://localhost:8060/dingtalk/webhook/send  
    send_resolved: true
# 抑制
# alertname、cluster、service相同的告警
# critical存在则warning的被抑制
inhibit_rules:
- equal: ['alertname', 'cluster', 'service']
  source_match:
    severity: 'critical'
  target_match:
    severity: 'warning'

4.3 创建服务

cat > /usr/lib/systemd/system/alertmanager.service <<EOF
[Unit]
Description=Alertmanager
After=network.target

[Service]
Type=simple
User=prometheus
ExecStart=/usr/local/alertmanager/alertmanager --web.external-url=http://example.com:9093 --config.file=/usr/local/alertmanager/alertmanager.yml --storage.path=/data/alertmanager/data
Restart=on-failure

[Install]
WantedBy=multi-user.target
EOF

4.4 启动服务

systemctl enable alertmanager.service
systemctl start alertmanager.service

5 PrometheusAlert

5.1 下载

mkdir -p /usr/local/prometheus/prometheusalert
wget https://github.com/feiyu563/PrometheusAlert/releases/download/v4.9.1/linux.zip
unzip linux.zip -d /usr/local/prometheus/prometheusalert

5.2 创建服务

cat > /usr/lib/systemd/system/prometheusalert.service <<'EOF'
[Unit]
Description=prometheusalert
After=network-online.target

[Service]
Restart=on-failure
WorkingDirectory=/usr/local/prometheusalert
ExecStart=/usr/local/prometheusalert/PrometheusAlert

[Install]
WantedBy=multi-user.target
EOF

5.3 修改配置

编辑prometheusalert/conf/app.conf，参考 https://feiyu563.gitbook.io/prometheusalert/conf

5.4 启动服务

systemctl enable alertmanager.service
systemctl start alertmanager.service

5.5 创建告警规则文件

推荐项目，里面有很多告警规则https://github.com/samber/awesome-prometheus-alerts

测试规则是不是正确

./promtool check rules rules/basic.yml

6 其他知识点

删除一些 job 任务或者 instance 的数据指标，则可以使用下面的命令：

curl -X POST -g 'http://localhost:9090/api/v1/admin/tsdb/delete_series?match[]={job="kubernetes"}'
curl -X POST -g 'http://localhost:9090/api/v1/admin/tsdb/delete_series?match[]={instance="10.244.2.158:9090"}'

参考：Prometheus 删除数据指标

赞赏支持

微信打赏

支付宝打赏