Kubernetes高可用性与灾难恢复配置：构建容错能力强的集群
Kubernetes高可用性与灾难恢复配置：构建容错能力强的集群
一、高可用性概述
Kubernetes高可用性是指集群在面对节点故障、网络中断等问题时能够保持服务正常运行的能力。
1.1 HA架构
┌─────────────────────────────────────────────────────────────────┐ │ 控制平面高可用 │ │ ┌──────────────┐ ┌──────────────┐ ┌──────────────┐ │ │ │ Master 1 │ │ Master 2 │ │ Master 3 │ │ │ │ API Server │ │ API Server │ │ API Server │ │ │ │ etcd │ │ etcd │ │ etcd │ │ │ │ Controller │ │ Controller │ │ Controller │ │ │ └──────┬───────┘ └──────┬───────┘ └──────┬───────┘ │ └─────────┼─────────────────┼─────────────────┼─────────────────┘ │ │ │ ▼ ▼ ▼ ┌─────────────────────────────────────────────────────────────────┐ │ Load Balancer │ └───────────────────────────┬─────────────────────────────────────┘ │ ┌─────────────────┼─────────────────┐ ▼ ▼ ▼ ┌─────────────────┐ ┌─────────────────┐ ┌─────────────────┐ │ Worker 1 │ │ Worker 2 │ │ Worker 3 │ │ ┌───────────┐ │ │ ┌───────────┐ │ │ ┌───────────┐ │ │ │ Pod A │ │ │ │ Pod B │ │ │ │ Pod C │ │ │ │ Pod D │ │ │ │ Pod E │ │ │ │ Pod F │ │ │ └───────────┘ │ │ └───────────┘ │ │ └───────────┘ │ └─────────────────┘ └─────────────────┘ └─────────────────┘1.2 HA组件
| 组件 | 高可用策略 |
|---|---|
| API Server | 多实例 + 负载均衡 |
| etcd | 分布式集群 |
| Controller Manager | 多实例选举 |
| Scheduler | 多实例选举 |
二、etcd高可用配置
2.1 etcd集群配置
apiVersion: v1 kind: Pod metadata: name: etcd-server spec: containers: - name: etcd image: quay.io/coreos/etcd:v3.5.0 command: - etcd - --name=etcd-0 - --initial-advertise-peer-urls=http://etcd-0:2380 - --listen-peer-urls=http://0.0.0.0:2380 - --listen-client-urls=http://0.0.0.0:2379 - --advertise-client-urls=http://etcd-0:2379 - --initial-cluster=etcd-0=http://etcd-0:2380,etcd-1=http://etcd-1:2380,etcd-2=http://etcd-2:2380 - --initial-cluster-state=new - --data-dir=/var/lib/etcd2.2 etcd备份配置
#!/bin/bash TIMESTAMP=$(date +%Y%m%d_%H%M%S) BACKUP_DIR="/backup/etcd" mkdir -p $BACKUP_DIR etcdctl snapshot save "$BACKUP_DIR/snapshot_$TIMESTAMP.db" etcdctl snapshot status "$BACKUP_DIR/snapshot_$TIMESTAMP.db"三、控制平面高可用配置
3.1 API Server配置
apiVersion: v1 kind: Service metadata: name: kubernetes spec: type: ClusterIP clusterIP: 10.96.0.1 ports: - port: 443 targetPort: 6443 selector: component: kube-apiserver3.2 Controller Manager配置
apiVersion: v1 kind: Pod metadata: name: kube-controller-manager spec: containers: - name: kube-controller-manager image: k8s.gcr.io/kube-controller-manager:v1.25.0 command: - kube-controller-manager - --leader-elect=true - --controllers=* - --cluster-name=my-cluster四、Worker节点高可用配置
4.1 Pod副本配置
apiVersion: apps/v1 kind: Deployment metadata: name: highly-available-app spec: replicas: 3 strategy: type: RollingUpdate rollingUpdate: maxSurge: 1 maxUnavailable: 0 template: spec: affinity: podAntiAffinity: requiredDuringSchedulingIgnoredDuringExecution: - labelSelector: matchExpressions: - key: app operator: In values: - my-app topologyKey: kubernetes.io/hostname4.2 Pod中断预算
apiVersion: policy/v1 kind: PodDisruptionBudget metadata: name: my-app-pdb spec: minAvailable: 2 selector: matchLabels: app: my-app五、灾难恢复配置
5.1 定期备份配置
apiVersion: batch/v1 kind: CronJob metadata: name: etcd-backup spec: schedule: "0 2 * * *" jobTemplate: spec: template: spec: containers: - name: etcd-backup image: quay.io/coreos/etcd:v3.5.0 command: - /bin/sh - -c - "etcdctl snapshot save /backup/snapshot.db" volumeMounts: - name: backup-volume mountPath: /backup volumes: - name: backup-volume persistentVolumeClaim: claimName: backup-pvc restartPolicy: OnFailure5.2 数据恢复配置
#!/bin/bash etcdctl snapshot restore /backup/snapshot.db \ --name=etcd-restore \ --initial-advertise-peer-urls=http://etcd-restore:2380 \ --initial-cluster=etcd-restore=http://etcd-restore:2380 \ --data-dir=/var/lib/etcd六、网络高可用配置
6.1 Ingress高可用
apiVersion: apps/v1 kind: Deployment metadata: name: nginx-ingress spec: replicas: 3 selector: matchLabels: app: nginx-ingress template: spec: containers: - name: nginx-ingress image: nginx/nginx-ingress:latest6.2 Service高可用
apiVersion: v1 kind: Service metadata: name: my-service spec: selector: app: my-app ports: - port: 80 targetPort: 8080 type: ClusterIP七、监控与告警配置
7.1 HA监控
apiVersion: monitoring.coreos.com/v1 kind: ServiceMonitor metadata: name: etcd-monitor spec: selector: matchLabels: app: etcd endpoints: - port: metrics interval: 30s7.2 HA告警规则
apiVersion: monitoring.coreos.com/v1 kind: PrometheusRule metadata: name: ha-alerts spec: groups: - name: etcd.rules rules: - alert: EtcdMembersDown expr: count(up{job="etcd"} == 0) > 0 for: 5m labels: severity: critical annotations: summary: "etcd member down"八、高可用性最佳实践
8.1 多区域部署
apiVersion: apps/v1 kind: Deployment metadata: name: multi-zone-app spec: replicas: 6 template: spec: affinity: podAntiAffinity: requiredDuringSchedulingIgnoredDuringExecution: - labelSelector: matchExpressions: - key: app operator: In values: - my-app topologyKey: topology.kubernetes.io/zone8.2 定期演练
#!/bin/bash echo "=== 开始高可用性演练 ===" kubectl cordon node-1 kubectl drain node-1 --ignore-daemonsets kubectl uncordon node-1 echo "=== 高可用性演练完成 ==="九、总结
高可用性配置需要关注：
- 控制平面HA：多Master节点部署
- 数据存储HA：etcd集群配置
- 应用HA：多副本+反亲和性配置
- 灾难恢复：定期备份和恢复演练
- 监控告警：及时发现和响应故障
建议建立完善的高可用体系，定期进行故障演练。
参考资料：
- Kubernetes HA文档
- etcd HA文档
- PodDisruptionBudget文档