当前位置：首页 > news >正文

云原生环境中的监控与可观测性最佳实践：从指标到追踪的全栈监控

news 2026/4/19 6:57:04

云原生环境中的监控与可观测性最佳实践：从指标到追踪的全栈监控

🔥 硬核开场

各位技术大佬们，今天咱们来聊聊云原生环境中的监控与可观测性。别跟我说你还在手动登录服务器看日志，那都2023年了！在云原生时代，监控是保障系统稳定运行的关键，可观测性是快速定位问题的利器。从指标监控到日志收集，从分布式追踪到告警管理，每一个环节都不能掉以轻心。今天susu就带你们从理论到实践，一步步构建云原生监控体系，全给你整明白！

📋 核心内容

1. 监控与可观测性的核心概念

什么是监控：收集系统的指标数据，监控系统的健康状态
什么是可观测性：通过收集和分析系统的指标、日志和追踪数据，理解系统的内部状态
可观测性三支柱：指标(Metrics)、日志(Logs)、追踪(Traces)

2. Prometheus：云原生监控的标准

Prometheus是云原生环境中最流行的监控系统，专为容器环境设计。

2.1 安装Prometheus

# 使用Helm安装Prometheus helm repo add prometheus-community https://prometheus-community.github.io/helm-charts helm repo update helm install prometheus prometheus-community/kube-prometheus-stack --namespace monitoring --create-namespace # 查看Prometheus组件状态 kubectl get pods -n monitoring

2.2 配置Prometheus监控目标

apiVersion: monitoring.coreos.com/v1 kind: ServiceMonitor metadata: name: app-monitor namespace: monitoring spec: selector: matchLabels: app: example-app endpoints: - port: metrics interval: 15s

2.3 查看Prometheus监控数据

# 端口转发 kubectl port-forward svc/prometheus-server -n monitoring 9090:9090 # 访问Prometheus UI # 打开浏览器访问 http://localhost:9090

3. Grafana：数据可视化的利器

Grafana是一个开源的数据可视化平台，可以与Prometheus等数据源集成，创建美观的监控Dashboard。

3.1 配置Grafana Dashboard

# 端口转发 kubectl port-forward svc/grafana -n monitoring 3000:3000 # 访问Grafana UI # 打开浏览器访问 http://localhost:3000 # 默认用户名：admin，密码：prom-operator

3.2 创建自定义Dashboard

登录Grafana UI
点击"+" -> "Dashboard"
点击"Add new panel"
选择Prometheus数据源
输入查询语句，如：rate(http_requests_total[5m])
配置面板标题和样式
保存Dashboard

4. Loki：日志收集与分析

Loki是Grafana Labs开发的日志聚合系统，专为云原生环境设计，与Prometheus和Grafana无缝集成。

4.1 安装Loki

# 使用Helm安装Loki helm repo add grafana https://grafana.github.io/helm-charts helm repo update helm install loki grafana/loki --namespace monitoring # 安装Promtail（日志收集器） helm install promtail grafana/promtail --namespace monitoring --set config.clients[0].url=http://loki:3100/loki/api/v1/push # 查看Loki组件状态 kubectl get pods -n monitoring

4.2 配置应用日志收集

apiVersion: apps/v1 kind: Deployment metadata: name: example-app namespace: default spec: replicas: 3 selector: matchLabels: app: example-app template: metadata: labels: app: example-app spec: containers: - name: example-app image: nginx:latest ports: - containerPort: 80 volumeMounts: - name: logs mountPath: /var/log/nginx volumes: - name: logs emptyDir: {}

4.3 查看日志

# 访问Grafana UI # 打开浏览器访问 http://localhost:3000 # 选择"Explore" -> 选择"Loki"数据源 # 输入查询语句，如：`{app="example-app"} |= "error"`

5. Jaeger：分布式追踪系统

Jaeger是一个开源的分布式追踪系统，可以帮助我们追踪请求在微服务架构中的流动路径，定位性能瓶颈。

5.1 安装Jaeger

# 使用Helm安装Jaeger helm repo add jaegertracing https://jaegertracing.github.io/helm-charts helm repo update helm install jaeger jaegertracing/jaeger --namespace monitoring # 查看Jaeger组件状态 kubectl get pods -n monitoring

5.2 配置应用追踪

apiVersion: apps/v1 kind: Deployment metadata: name: example-app namespace: default spec: replicas: 3 selector: matchLabels: app: example-app template: metadata: labels: app: example-app spec: containers: - name: example-app image: nginx:latest ports: - containerPort: 80 env: - name: JAEGER_AGENT_HOST value: "jaeger-agent.monitoring.svc.cluster.local" - name: JAEGER_AGENT_PORT value: "6831"

5.3 查看追踪数据

# 端口转发 kubectl port-forward svc/jaeger-query -n monitoring 16686:16686 # 访问Jaeger UI # 打开浏览器访问 http://localhost:16686

6. 告警管理

6.1 配置Prometheus告警规则

apiVersion: monitoring.coreos.com/v1 kind: PrometheusRule metadata: name: app-alerts namespace: monitoring spec: groups: - name: example-app rules: - alert: HighCPUUsage expr: 100 - (avg by(instance) (irate(node_cpu_seconds_total{mode="idle"}[5m])) * 100) > 80 for: 5m labels: severity: warning annotations: summary: "High CPU Usage" description: "CPU usage is above 80% for 5 minutes" - alert: HighMemoryUsage expr: (node_memory_MemTotal_bytes - node_memory_MemAvailable_bytes) / node_memory_MemTotal_bytes * 100 > 80 for: 5m labels: severity: warning annotations: summary: "High Memory Usage" description: "Memory usage is above 80% for 5 minutes"

6.2 配置Alertmanager

apiVersion: monitoring.coreos.com/v1 kind: Alertmanager metadata: name: prometheus-alertmanager namespace: monitoring spec: replicas: 3 alertmanagerConfigSelector: matchLabels: alertmanagerConfig: example resources: requests: cpu: 100m memory: 128Mi limits: cpu: 500m memory: 256Mi --- apiVersion: monitoring.coreos.com/v1 kind: AlertmanagerConfig metadata: name: example namespace: monitoring labels: alertmanagerConfig: example spec: receivers: - name: email emailConfigs: - to: "admin@example.com" from: "alertmanager@example.com" smarthost: "smtp.example.com:587" authUsername: "alertmanager" authPassword: name: smtp-secret key: password route: groupBy: - alertname - cluster - service groupInterval: 5m groupWait: 30s repeatInterval: 1h receiver: email routes: - match: severity: critical receiver: email