当前位置：首页 > news >正文

Kubernetes 与大数据集成最佳实践

news 2026/6/8 1:54:30

Kubernetes 与大数据集成最佳实践

一、前言

哥们，别整那些花里胡哨的。大数据工作负载在 Kubernetes 中越来越常见，今天直接上硬货，教你如何在 Kubernetes 中集成和管理大数据工作负载。

二、大数据工作负载类型

类型	适用场景	优势	劣势
Hadoop	批处理	成熟稳定	资源消耗大
Spark	流处理	高性能	配置复杂
Kafka	消息队列	高吞吐	存储需求大
Flink	实时处理	低延迟	学习成本高
HBase	列式存储	高并发	运维复杂

三、实战配置

1. Hadoop 配置

apiVersion: apps/v1 kind: StatefulSet metadata: name: hadoop-namenode namespace: bigdata spec: serviceName: hadoop-namenode replicas: 1 selector: matchLabels: app: hadoop-namenode template: metadata: labels: app: hadoop-namenode spec: containers: - name: namenode image: apache/hadoop:3.3.4 command: - /bin/bash - -c - | hdfs namenode -format hdfs namenode ports: - containerPort: 9870 - containerPort: 9000 volumeMounts: - name: namenode-data mountPath: /hadoop/dfs/name volumeClaimTemplates: - metadata: name: namenode-data spec: accessModes: ["ReadWriteOnce"] resources: requests: storage: 100Gi storageClassName: high-performance --- apiVersion: apps/v1 kind: StatefulSet metadata: name: hadoop-datanode namespace: bigdata spec: serviceName: hadoop-datanode replicas: 3 selector: matchLabels: app: hadoop-datanode template: metadata: labels: app: hadoop-datanode spec: containers: - name: datanode image: apache/hadoop:3.3.4 command: - /bin/bash - -c - | hdfs datanode ports: - containerPort: 9864 volumeMounts: - name: datanode-data mountPath: /hadoop/dfs/data volumeClaimTemplates: - metadata: name: datanode-data spec: accessModes: ["ReadWriteOnce"] resources: requests: storage: 200Gi storageClassName: high-performance

2. Spark 配置

apiVersion: apps/v1 kind: Deployment metadata: name: spark-master namespace: bigdata spec: replicas: 1 selector: matchLabels: app: spark-master template: metadata: labels: app: spark-master spec: containers: - name: spark-master image: bitnami/spark:3.3.1 env: - name: SPARK_MODE value: "master" ports: - containerPort: 7077 - containerPort: 8080 --- apiVersion: apps/v1 kind: Deployment metadata: name: spark-worker namespace: bigdata spec: replicas: 3 selector: matchLabels: app: spark-worker template: metadata: labels: app: spark-worker spec: containers: - name: spark-worker image: bitnami/spark:3.3.1 env: - name: SPARK_MODE value: "worker" - name: SPARK_MASTER_URL value: "spark://spark-master:7077" ports: - containerPort: 8081 resources: requests: cpu: "2" memory: "4Gi" limits: cpu: "4" memory: "8Gi"

3. Kafka 配置

apiVersion: apps/v1 kind: StatefulSet metadata: name: kafka namespace: bigdata spec: serviceName: kafka replicas: 3 selector: matchLabels: app: kafka template: metadata: labels: app: kafka spec: containers: - name: kafka image: bitnami/kafka:3.2.3 env: - name: KAFKA_ZOOKEEPER_CONNECT value: "zookeeper:2181" - name: KAFKA_ADVERTISED_LISTENERS value: "PLAINTEXT://kafka:9092" - name: KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR value: "3" ports: - containerPort: 9092 volumeMounts: - name: kafka-data mountPath: /bitnami/kafka volumeClaimTemplates: - metadata: name: kafka-data spec: accessModes: ["ReadWriteOnce"] resources: requests: storage: 100Gi storageClassName: high-performance --- apiVersion: apps/v1 kind: StatefulSet metadata: name: zookeeper namespace: bigdata spec: serviceName: zookeeper replicas: 3 selector: matchLabels: app: zookeeper template: metadata: labels: app: zookeeper spec: containers: - name: zookeeper image: bitnami/zookeeper:3.7.0 env: - name: ZOO_REPLICAS value: "3" ports: - containerPort: 2181 volumeMounts: - name: zookeeper-data mountPath: /bitnami/zookeeper volumeClaimTemplates: - metadata: name: zookeeper-data spec: accessModes: ["ReadWriteOnce"] resources: requests: storage: 50Gi storageClassName: high-performance

4. Flink 配置

apiVersion: apps/v1 kind: Deployment metadata: name: flink-jobmanager namespace: bigdata spec: replicas: 1 selector: matchLabels: app: flink-jobmanager template: metadata: labels: app: flink-jobmanager spec: containers: - name: jobmanager image: flink:1.16.0 command: - /bin/bash - -c - | /opt/flink/bin/jobmanager.sh start-foreground ports: - containerPort: 8081 - containerPort: 6123 resources: requests: cpu: "1" memory: "2Gi" limits: cpu: "2" memory: "4Gi" --- apiVersion: apps/v1 kind: Deployment metadata: name: flink-taskmanager namespace: bigdata spec: replicas: 3 selector: matchLabels: app: flink-taskmanager template: metadata: labels: app: flink-taskmanager spec: containers: - name: taskmanager image: flink:1.16.0 command: - /bin/bash - -c - | /opt/flink/bin/taskmanager.sh start-foreground env: - name: JOB_MANAGER_RPC_ADDRESS value: "flink-jobmanager" resources: requests: cpu: "2" memory: "4Gi" limits: cpu: "4" memory: "8Gi"

四、大数据工作负载优化

1. 资源管理

apiVersion: v1 kind: ResourceQuota metadata: name: bigdata-quota namespace: bigdata spec: hard: requests.cpu: "20" requests.memory: "40Gi" limits.cpu: "40" limits.memory: "80Gi" pods: "50" --- apiVersion: v1 kind: LimitRange metadata: name: bigdata-limits namespace: bigdata spec: limits: - default: cpu: "2" memory: "4Gi" defaultRequest: cpu: "1" memory: "2Gi" type: Container

2. 存储优化

apiVersion: storage.k8s.io/v1 kind: StorageClass metadata: name: bigdata-storage provisioner: kubernetes.io/aws-ebs parameters: type: io2 iopsPerGB: "5000" throughput: "1000" reclaimPolicy: Retain allowVolumeExpansion: true volumeBindingMode: WaitForFirstConsumer

3. 网络优化

apiVersion: networking.k8s.io/v1 kind: NetworkPolicy metadata: name: bigdata-network-policy namespace: bigdata spec: podSelector: matchLabels: app: hadoop policyTypes: - Ingress - Egress ingress: - from: - podSelector: matchLabels: app: spark ports: - protocol: TCP port: 9000 egress: - to: - podSelector: matchLabels: app: hadoop - podSelector: matchLabels: app: spark