当前位置：首页 > news >正文

Kubernetes监控与可观测性深度解析与实践

news 2026/5/9 11:51:30

Kubernetes监控与可观测性深度解析与实践

监控与可观测性概述

在Kubernetes环境中，监控与可观测性是确保应用稳定运行的关键。本文将深入探讨Kubernetes监控体系的核心组件、配置方法和最佳实践。

监控体系架构

┌─────────────────────────────────────────────────────────────────┐ │ Kubernetes监控架构 │ ├─────────────────────────────────────────────────────────────────┤ │ ┌─────────────┐ ┌─────────────┐ ┌─────────────┐ │ │ │ Metrics │ │ Logs │ │ Tracing │ │ │ │ (Prometheus)│ │ (ELK/Loki) │ │ (Jaeger) │ │ │ └─────────────┘ └─────────────┘ └─────────────┘ │ │ │ │ │ │ │ ▼ ▼ ▼ │ │ ┌─────────────────────────────────────────────────────────┐ │ │ │ Grafana │ │ │ │ (统一可视化平台) │ │ │ └─────────────────────────────────────────────────────────┘ │ │ │ │ │ ▼ │ │ ┌─────────────────────────────────────────────────────────┐ │ │ │ Alertmanager │ │ │ │ (告警管理与通知) │ │ │ └─────────────────────────────────────────────────────────┘ │ └─────────────────────────────────────────────────────────────────┘

Prometheus监控

1. Prometheus配置

apiVersion: monitoring.coreos.com/v1 kind: Prometheus metadata: name: prometheus namespace: monitoring spec: replicas: 2 serviceAccountName: prometheus serviceMonitorSelector: matchLabels: team: frontend resources: requests: memory: 400Mi enableAdminAPI: false

2. ServiceMonitor配置

apiVersion: monitoring.coreos.com/v1 kind: ServiceMonitor metadata: name: myapp-monitor namespace: monitoring spec: selector: matchLabels: app: myapp endpoints: - port: http interval: 30s path: /metrics scrapeTimeout: 10s

3. 自定义指标

apiVersion: v1 kind: Service metadata: name: myapp-metrics labels: app: myapp spec: selector: app: myapp ports: - name: http port: 8080 targetPort: 8080

Grafana可视化

1. Grafana配置

apiVersion: grafana.integreatly.org/v1beta1 kind: Grafana metadata: name: grafana namespace: monitoring spec: deployment: spec: replicas: 1 config: log: mode: "console" security: admin_user: admin admin_password: secret

2. 数据源配置

apiVersion: grafana.integreatly.org/v1beta1 kind: GrafanaDataSource metadata: name: prometheus-datasource namespace: monitoring spec: name: prometheus type: prometheus access: proxy url: http://prometheus-k8s.monitoring.svc.cluster.local:9090 basicAuth: false

3. 仪表板配置

apiVersion: grafana.integreatly.org/v1beta1 kind: GrafanaDashboard metadata: name: kubernetes-dashboard namespace: monitoring spec: name: kubernetes-dashboard.json folder: Kubernetes datasources: - name: Prometheus inputName: DS_PROMETHEUS

日志管理

1. Loki配置

apiVersion: loki.grafana.com/v1 kind: Loki metadata: name: loki namespace: monitoring spec: size: 1x.small storage: secret: type: s3 bucketNames: - loki-data endpoint: s3.amazonaws.com region: us-west-2

2. Fluentd配置

apiVersion: v1 kind: ConfigMap metadata: name: fluentd-config namespace: logging data: fluent.conf: | <source> @type tail path /var/log/containers/*.log pos_file /var/log/fluentd-containers.log.pos tag kubernetes.* read_from_head true <parse> @type json time_key time time_format %Y-%m-%dT%H:%M:%S.%NZ </parse> </source> <match kubernetes.**> @type loki url http://loki.monitoring.svc.cluster.local:3100 flush_interval 10s </match>

3. EFK Stack配置

apiVersion: elasticsearch.k8s.elastic.co/v1 kind: Elasticsearch metadata: name: quickstart namespace: logging spec: version: 8.8.0 nodeSets: - name: default count: 3 config: node.store.allow_mmap: false

分布式追踪

1. Jaeger配置

apiVersion: jaegertracing.io/v1 kind: Jaeger metadata: name: jaeger namespace: tracing spec: strategy: production storage: type: elasticsearch options: es: server-urls: http://elasticsearch:9200 ingress: enabled: true

2. OpenTelemetry配置

apiVersion: opentelemetry.io/v1alpha1 kind: OpenTelemetryCollector metadata: name: my-collector namespace: monitoring spec: config: | receivers: otlp: protocols: grpc: http: jaeger: protocols: grpc: thrift_http: zipkin: processors: batch: exporters: otlp: endpoint: jaeger-collector:4317 tls: insecure: true service: pipelines: traces: receivers: [otlp, jaeger, zipkin] processors: [batch] exporters: [otlp]

告警管理

1. Alertmanager配置

apiVersion: monitoring.coreos.com/v1 kind: Alertmanager metadata: name: alertmanager namespace: monitoring spec: replicas: 3 serviceAccountName: alertmanager config: global: resolve_timeout: 5m route: group_by: ['alertname'] group_wait: 10s group_interval: 10s repeat_interval: 1h receiver: 'webhook' receivers: - name: 'webhook' webhook_configs: - url: 'http://alertmanager-webhook:8080/'

2. 告警规则

apiVersion: monitoring.coreos.com/v1 kind: PrometheusRule metadata: name: kubernetes-rules namespace: monitoring spec: groups: - name: kubernetes.rules rules: - alert: HighPodMemoryUsage expr: sum(container_memory_usage_bytes{namespace!="kube-system"}) / sum(kube_pod_resource_limits_memory_bytes{namespace!="kube-system"}) > 0.8 for: 5m labels: severity: warning annotations: summary: "High memory usage in namespace {{ $labels.namespace }}" description: "Memory usage is {{ $value | humanizePercentage }}"

监控最佳实践

1. 指标采集策略

apiVersion: monitoring.coreos.com/v1 kind: ServiceMonitor metadata: name: optimized-monitor namespace: monitoring spec: selector: matchLabels: app: myapp endpoints: - port: http interval: 30s path: /metrics relabelings: - sourceLabels: [__meta_kubernetes_pod_name] targetLabel: pod replacement: $1

2. 资源监控

apiVersion: v1 kind: ConfigMap metadata: name: resource-metrics-config namespace: kube-system data: config.yaml: | kubelet: metricRelabelings: - action: keep sourceLabels: ["__name__"] regex: "kubelet_running_pods|kubelet_node_name"

3. 日志分级

apiVersion: v1 kind: ConfigMap metadata: name: log-level-config data: log4j2.xml: | <Configuration status="INFO"> <Appenders> <Console name="Console" target="SYSTEM_OUT"> <PatternLayout pattern="%d{HH:mm:ss.SSS} [%t] %-5level %logger{36} - %msg%n"/> </Console> </Appenders> <Loggers> <Root level="INFO"> <AppenderRef ref="Console"/> </Root> <Logger name="com.example" level="DEBUG"/> </Loggers> </Configuration>

可观测性工具链

工具	功能	用途
Prometheus	指标采集与存储	监控系统指标
Grafana	可视化仪表板	展示监控数据
Loki	日志聚合	存储和查询日志
Jaeger	分布式追踪	追踪请求链路
OpenTelemetry	统一可观测性	收集指标、日志和追踪
Alertmanager	告警管理	处理和发送告警

实战案例：构建完整的可观测性平台

架构设计

┌─────────────────────────────────────────────────────────────────┐ │ 可观测性平台架构 │ ├─────────────────────────────────────────────────────────────────┤ │ ┌─────────────┐ ┌─────────────┐ ┌─────────────┐ │ │ │ Metrics │ │ Logs │ │ Tracing │ │ │ │ Server │ │ Collector │ │ Agent │ │ │ └─────────────┘ └─────────────┘ └─────────────┘ │ │ │ │ │ │ │ ▼ ▼ ▼ │ │ ┌─────────────────────────────────────────────────────────┐ │ │ │ Storage Layer │ │ │ │ Prometheus │ Loki │ Elasticsearch │ Jaeger │ │ │ └─────────────────────────────────────────────────────────┘ │ │ │ │ │ ▼ │ │ ┌─────────────────────────────────────────────────────────┐ │ │ │ Visualization │ │ │ │ Grafana Dashboard │ │ │ └─────────────────────────────────────────────────────────┘ │ │ │ │ │ ▼ │ │ ┌─────────────────────────────────────────────────────────┐ │ │ │ Alerting │ │ │ │ Alertmanager + Webhooks │ │ │ └─────────────────────────────────────────────────────────┘ │ └─────────────────────────────────────────────────────────────────┘