Prometheus介绍及监控平台部署

1. 核心架构概览

plaintext

┌─────────────────────────────────────────────────────────────────┐ │ Prometheus 架构 │ ├─────────────────────────────────────────────────────────────────┤ │ ┌────────┐ ┌────────┐ ┌────────┐ ┌────────┐ │ │ │Exporter│ │Exporter│ │Exporter│ │Exporter│ (:9100/9104) │ │ └───┬────┘ └───┬────┘ └───┬────┘ └───┬────┘ │ │ └───────────┴───────────┴───────────┘ │ │ │ Pull (15s) │ │ ▼ │ │ ┌──────────────┐ │ │ │ Prometheus │──┐ │ │ │ Server │ │ ┌──────────────┐ │ │ │ ┌──────────┐ │ └───▶│ Alertmanager │──▶通知 │ │ │ │ TSDB │ │ └──────────────┘ │ │ │ └──────────┘ │ │ │ └──────┬───────┘ │ │ ▼ │ │ ┌──────────────┐ (:3000) │ │ │ Grafana │ │ │ └──────────────┘ │ └─────────────────────────────────────────────────────────────────┘

流程：Exporter暴露/metrics→ Prometheus定时Pull → TSDB存储 → Alertmanager告警 → Grafana展示

2. 部署安装（Docker Compose）

yaml

# docker-compose.yml version: '3.8' services: prometheus: image: prom/prometheus:v2.47.0 container_name: prometheus restart: unless-stopped ports: - "9090:9090" volumes: - ./prometheus/prometheus.yml:/etc/prometheus/prometheus.yml - ./prometheus/rules:/etc/prometheus/rules - prometheus_data:/prometheus command: - '--config.file=/etc/prometheus/prometheus.yml' - '--storage.tsdb.path=/prometheus' - '--storage.tsdb.retention.time=15d' - '--web.enable-lifecycle' networks: - monitoring alertmanager: image: prom/alertmanager:v0.26.0 container_name: alertmanager restart: unless-stopped ports: - "9093:9093" volumes: - ./alertmanager/alertmanager.yml:/etc/alertmanager/alertmanager.yml networks: - monitoring grafana: image: grafana/grafana:10.1.0 container_name: grafana restart: unless-stopped ports: - "3000:3000" environment: - GF_SECURITY_ADMIN_USER=admin - GF_SECURITY_ADMIN_PASSWORD=admin123 volumes: - grafana_data:/var/lib/grafana networks: - monitoring node-exporter: image: prom/node-exporter:v1.6.1 container_name: node-exporter restart: unless-stopped ports: - "9100:9100" command: - '--path.procfs=/host/proc' - '--path.sysfs=/host/sys' - '--path.rootfs=/host' - '--collector.filesystem.mount-points-exclude=^/(sys|proc|dev|host|etc)($$|$)' volumes: - /proc:/host/proc:ro - /sys:/host/sys:ro - /:/host:ro networks: - monitoring networks: monitoring: driver: bridge volumes: prometheus_data: grafana_data:

3. 核心配置（prometheus.yml详解）

yaml

# prometheus.yml global: scrape_interval: 15s evaluation_interval: 15s external_labels: cluster: 'prod' alerting: alertmanagers: - static_configs: - targets: ['alertmanager:9093'] rule_files: - "rules/*.yml" scrape_configs: - job_name: 'prometheus' static_configs: - targets: ['localhost:9090'] - job_name: 'node' static_configs: - targets: ['node-exporter:9100'] labels: env: 'prod' - job_name: 'file_sd' file_sd_configs: - files: - 'targets/*.json' refresh_interval: 30s - job_name: 'kubernetes-pods' kubernetes_sd_configs: - role: pod relabel_configs: - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape] action: keep regex: true - job_name: 'relabel_demo' static_configs: - targets: ['192.168.1.100:8080'] relabel_configs: - source_labels: [__address__] regex: '([^:]+):(\d+)' target_label: instance replacement: '${1}' - target_label: env replacement: 'prod' - regex: '__meta_.*' action: labeldrop

4. Exporter部署

node_exporter安装

bash

wget https://github.com/prometheus/node_exporter/releases/download/v1.6.1/node_exporter-1.6.1.linux-amd64.tar.gz tar xzf node_exporter-1.6.1.linux-amd64.tar.gz sudo mv node_exporter-1.6.1.linux-amd64/node_exporter /usr/local/bin/ sudo tee /etc/systemd/system/node-exporter.service <<EOF [Unit] Description=Node Exporter After=network.target [Service] ExecStart=/usr/local/bin/node_exporter Restart=on-failure [Install] WantedBy=multi-user.target EOF sudo systemctl daemon-reload && sudo systemctl enable --now node-exporter

常用Exporter

表格

5. PromQL查询基础

promql

# 即时向量 100 - (avg by (instance) (rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100) # CPU使用率 100 * (1 - (node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes)) # 内存使用率 # 区间向量 rate(node_cpu_seconds_total{mode="user"}[5m]) # 变化率 increase(http_requests_total[1h]) # 增量 # 聚合 sum by (instance, job) (rate(node_cpu_seconds_total[5m])) count(node_cpu_seconds_total) max by (service) (http_request_duration_seconds_bucket) # 函数 predict_linear(node_filesystem_free_bytes{mountpoint="/"}[1h], 4*3600) # 预测 irate(node_cpu_seconds_total{mode="user"}[5m]) # 瞬时变化率 label_replace(up{job="node"}, "hostname", "$1", "instance", "([^:]+):.*")

6. Alertmanager告警配置

告警规则

yaml

# rules/alerts.yml groups: - name: node_alerts rules: - alert: NodeDown expr: up{job="node"} == 0 for: 1m labels: severity: critical annotations: summary: "节点 {{ $labels.instance }} 宕机" - alert: HighCPU expr: 100 - (avg by (instance) (rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100) > 80 for: 5m labels: severity: warning annotations: summary: "CPU使用率超过80%，当前: {{ $value | printf \"%.2f\" }}%" - alert: LowMemory expr: (node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes) < 0.1 for: 3m labels: severity: warning - alert: DiskSpaceLow expr: (node_filesystem_avail_bytes{mountpoint="/"} / node_filesystem_size_bytes{mountpoint="/"}) < 0.1 for: 2m labels: severity: critical

Alertmanager配置

yaml

# alertmanager/alertmanager.yml global: resolve_timeout: 5m smtp_smarthost: 'smtp.qq.com:587' smtp_from: 'alert@example.com' smtp_auth_password: 'xxxxxx' route: group_by: ['alertname', 'cluster'] group_wait: 30s group_interval: 5m repeat_interval: 4h receiver: 'default-receiver' routes: - match: severity: critical receiver: 'critical-receiver' group_wait: 10s receivers: - name: 'default-receiver' email_configs: - to: 'ops@example.com' send_resolved: true slack_configs: - channel: '#alerts' send_resolved: true - name: 'critical-receiver' webhook_configs: - url: 'http://dingtalk:8060/dingtalk/webhook' inhibit_rules: - source_match: severity: 'critical' target_match: severity: 'warning' equal: ['instance']

7. 常用命令与API

bash

# 热加载配置 curl -X POST http://localhost:9090/-/reload # TSDB操作 curl http://localhost:9090/api/v1/status/tsdb curl -X POST -g 'http://localhost:9090/api/v1/admin/tsdb/delete_series?match[]={job="test"}' curl -X POST http://localhost:9090/api/v1/admin/tsdb/clean_tombstones # HTTP API curl -G http://localhost:9090/api/v1/query --data-urlencode 'query=up{job="node"}' curl -G http://localhost:9090/api/v1/query_range \ --data-urlencode 'query=up{job="node"}' \ --data-urlencode 'start=2024-01-01T00:00:00Z' \ --data-urlencode 'end=2024-01-01T01:00:00Z' \ --data-urlencode 'step=60s' curl http://localhost:9090/api/v1/targets curl http://localhost:9090/api/v1/alerts curl http://localhost:9090/api/v1/rules

8. 常见问题排查

问题1: Target Down

bash

curl http://localhost:9090/api/v1/targets | jq '.data.activeTargets[] | select(.health=="down")' nc -zv <target_ip> <port> docker logs node-exporter curl http://<target>:9100/metrics

问题2: 指标缺失

bash

curl http://localhost:9090/api/v1/label/__name__/values | jq '.data[]' | grep -i <metric> curl -G http://localhost:9090/api/v1/query --data-urlencode 'query=<metric_name>_total' curl http://localhost:9090/api/v1/targets | jq '.data.activeTargets[].labels'

问题3: 告警不触发

bash

curl http://localhost:9090/api/v1/rules | jq '.data.groups[].rules[] | select(.type=="alerting")' curl -s http://localhost:9093/api/v1/status curl http://localhost:9093/api/v1/silences

表格

9. 最佳实践

命名规范

yaml

# 指标名: <域>_<子系统>_<名称>_<单位> node_memory_Available_bytes http_request_duration_seconds # 标签: app_name, env, region, cluster, instance # 避免高基数标签（user_id, ip等）

联邦集群

yaml

- job_name: 'federate' metrics_path: '/federate' params: 'match[]': ['{__name__=~".+"}'] static_configs: - targets: - 'prometheus-prod:9090' - 'prometheus-prod2:9090'

高可用

plaintext

┌─────────────┐ ┌─────────────┐ │ Prometheus │ │ Prometheus │ # 双写 │ Primary │ │ Replica │ └──────┬──────┘ └──────┬──────┘ └────────┬─────────┘ ▼ ┌──────────────┐ │Thanos Receiver│ # 统一存储 └──────────────┘

远程存储

yaml

remote_write: - url: http://thanos-receive:19291/api/v1/receive queue_config: capacity: 10000 max_shards: 30 remote_read: - url: http://thanos-query:10912/api/v1/read read_recent: true

性能优化

1. 标签基数控制: 避免超过10万标签组合
2. 抓取间隔: 高频5s，低频60s
3. 记录规则: 预聚合复杂查询
4. 存储清理: 合理保留周期
5. 联邦分区: 按服务域拆分Prometheus