当前位置：首页 > news >正文

Kubernetes机器学习平台搭建：构建企业级ML训练环境

news 2026/5/24 23:19:29

Kubernetes机器学习平台搭建：构建企业级ML训练环境

一、机器学习平台概述

Kubernetes机器学习平台是基于K8s构建的ML训练和部署基础设施，支持数据科学家进行模型训练、验证和部署。

1.1 ML平台架构

┌─────────────────────────┐ │ 用户界面 │ │ (Jupyter/TensorBoard) │ └───────────┬─────────────┘ │ ┌─────────────────────────┼─────────────────────────┐ │ │ │ ▼ ▼ ▼ ┌───────────────┐ ┌───────────────┐ ┌───────────────┐ │ 训练调度器 │ │ 模型仓库 │ │ 数据存储 │ │ (Kubeflow) │ │ (MLflow) │ │ (MinIO) │ └───────────────┘ └───────────────┘ └───────────────┘ │ │ │ ▼ ▼ ▼ ┌───────────────┐ ┌───────────────┐ ┌───────────────┐ │ GPU节点池 │ │ CPU节点池 │ │ 存储集群 │ │ (训练任务) │ │ (预处理) │ │ (数据/模型) │ └───────────────┘ └───────────────┘ └───────────────┘

1.2 核心组件

组件	功能	工具
训练调度	管理训练任务	Kubeflow、Argo Workflows
模型管理	模型版本控制	MLflow、DVC
数据存储	数据集管理	MinIO、PV/PVC
资源管理	GPU/CPU调度	Kubernetes调度器
可视化	实验追踪	TensorBoard、Weights & Biases

二、Kubeflow部署

2.1 Kubeflow安装

# 安装Kubeflow export KUBEFLOW_RELEASE_VERSION=v1.8.0 export KUSTOMIZE_VERSION=v5.0.1 git clone https://github.com/kubeflow/manifests.git cd manifests git checkout ${KUBEFLOW_RELEASE_VERSION} # 部署Kubeflow while ! kustomize build example | kubectl apply -f -; do echo "Retrying..."; sleep 10; done

2.2 Kubeflow Pipeline配置

apiVersion: kubeflow.org/v1 kind: Pipeline metadata: name: ml-pipeline spec: pipelineSpec: tasks: - name: preprocess taskSpec: podSpec: containers: - name: preprocess image: preprocess:latest command: ["python", "preprocess.py"] - name: train taskSpec: podSpec: containers: - name: train image: train:latest command: ["python", "train.py"] resources: limits: nvidia.com/gpu: 1 dependencies: - preprocess

三、MLflow配置

3.1 MLflow部署

apiVersion: apps/v1 kind: Deployment metadata: name: mlflow namespace: mlflow spec: replicas: 1 selector: matchLabels: app: mlflow template: metadata: labels: app: mlflow spec: containers: - name: mlflow image: mlflow:latest ports: - containerPort: 5000 env: - name: MLFLOW_S3_ENDPOINT_URL value: http://minio:9000 - name: AWS_ACCESS_KEY_ID valueFrom: secretKeyRef: name: minio-creds key: accesskey - name: AWS_SECRET_ACCESS_KEY valueFrom: secretKeyRef: name: minio-creds key: secretkey command: - mlflow - server - --host=0.0.0.0 - --port=5000 - --backend-store-uri=postgresql://mlflow:password@postgres/mlflow - --default-artifact-root=s3://mlflow/

3.2 MLflow模型注册

import mlflow import mlflow.sklearn mlflow.set_tracking_uri("http://mlflow:5000") with mlflow.start_run(): # 训练模型 model = train_model() # 记录参数 mlflow.log_param("learning_rate", 0.01) # 记录指标 mlflow.log_metric("accuracy", 0.95) # 保存模型 mlflow.sklearn.log_model(model, "model") # 注册模型 mlflow.register_model( "runs:/{}/model".format(mlflow.active_run().info.run_id), "my-model" )

四、GPU资源管理

4.1 GPU节点配置

apiVersion: v1 kind: Node metadata: name: gpu-node-01 labels: nvidia.com/gpu.present: "true" node-role.kubernetes.io/gpu: "" spec: taints: - key: nvidia.com/gpu value: "true" effect: NoSchedule

4.2 GPU资源请求

apiVersion: v1 kind: Pod metadata: name: gpu-training-pod spec: tolerations: - key: nvidia.com/gpu operator: Equal value: "true" effect: NoSchedule containers: - name: training image: tensorflow/tensorflow:latest-gpu command: ["python", "train.py"] resources: limits: nvidia.com/gpu: 2 memory: 32Gi cpu: "8" requests: nvidia.com/gpu: 2 memory: 16Gi cpu: "4"

五、数据存储配置

5.1 MinIO部署

apiVersion: apps/v1 kind: StatefulSet metadata: name: minio namespace: minio spec: serviceName: minio replicas: 4 selector: matchLabels: app: minio template: metadata: labels: app: minio spec: containers: - name: minio image: minio/minio:latest ports: - containerPort: 9000 command: - minio - server - /data - --console-address - ":9001" volumeMounts: - name: data mountPath: /data env: - name: MINIO_ROOT_USER valueFrom: secretKeyRef: name: minio-creds key: accesskey - name: MINIO_ROOT_PASSWORD valueFrom: secretKeyRef: name: minio-creds key: secretkey volumeClaimTemplates: - metadata: name: data spec: accessModes: ["ReadWriteOnce"] resources: requests: storage: 100Gi

5.2 PVC配置

apiVersion: v1 kind: PersistentVolumeClaim metadata: name: ml-data namespace: ml spec: accessModes: - ReadWriteMany resources: requests: storage: 500Gi storageClassName: nfs-storage

六、JupyterHub部署

6.1 JupyterHub配置

apiVersion: hub.jupyter.org/v1 kind: Hub metadata: name: jupyterhub namespace: jupyterhub spec: image: name: jupyterhub/k8s-hub tag: 2.0.0 proxy: secretToken: <secret-token> auth: type: github github: clientId: <client-id> clientSecret: <client-secret> callbackUrl: https://jupyter.example.com/hub/oauth_callback singleuser: image: name: jupyter/scipy-notebook tag: latest storage: type: persistent-claim capacity: 10Gi

6.2 用户配置

apiVersion: hub.jupyter.org/v1 kind: User metadata: name: datascientist namespace: jupyterhub spec: profile: displayName: Data Scientist admin: false server: resources: limits: cpu: "4" memory: 16Gi requests: cpu: "2" memory: 8Gi

七、TensorBoard配置

7.1 TensorBoard部署

apiVersion: v1 kind: Service metadata: name: tensorboard namespace: ml spec: type: ClusterIP selector: app: tensorboard ports: - port: 6006 targetPort: 6006 --- apiVersion: apps/v1 kind: Deployment metadata: name: tensorboard namespace: ml spec: replicas: 1 selector: matchLabels: app: tensorboard template: metadata: labels: app: tensorboard spec: containers: - name: tensorboard image: tensorflow/tensorflow:latest command: - tensorboard - --logdir=/logs - --host=0.0.0.0 ports: - containerPort: 6006 volumeMounts: - name: logs mountPath: /logs volumes: - name: logs persistentVolumeClaim: claimName: tensorboard-logs

八、模型部署

8.1 TensorFlow Serving

apiVersion: v1 kind: Service metadata: name: tf-serving namespace: ml spec: type: ClusterIP selector: app: tf-serving ports: - port: 8501 targetPort: 8501 --- apiVersion: apps/v1 kind: Deployment metadata: name: tf-serving namespace: ml spec: replicas: 3 selector: matchLabels: app: tf-serving template: metadata: labels: app: tf-serving spec: containers: - name: tf-serving image: tensorflow/serving:latest ports: - containerPort: 8500 - containerPort: 8501 args: - "--model_name=my-model" - "--model_base_path=s3://models/my-model" env: - name: AWS_ACCESS_KEY_ID valueFrom: secretKeyRef: name: minio-creds key: accesskey - name: AWS_SECRET_ACCESS_KEY valueFrom: secretKeyRef: name: minio-creds key: secretkey - name: S3_ENDPOINT value: http://minio:9000

8.2 gRPC推理服务

apiVersion: v1 kind: Service metadata: name: model-service namespace: ml spec: type: ClusterIP selector: app: model-service ports: - port: 9000 targetPort: 9000 name: grpc - port: 8080 targetPort: 8080 name: http

九、监控与日志

9.1 训练指标监控

apiVersion: monitoring.coreos.com/v1 kind: ServiceMonitor metadata: name: ml-monitor namespace: monitoring spec: selector: matchLabels: app: ml-exporter endpoints: - port: metrics interval: 15s

9.2 资源使用监控

apiVersion: v1 kind: ConfigMap metadata: name: ml-metrics-config namespace: monitoring data: prometheus.rules: | groups: - name: ml.rules rules: - record: ml_training_duration_seconds expr: sum(rate(kube_pod_running_duration_seconds{app="training"}[5m])) - record: ml_gpu_utilization expr: sum(nvidia_gpu_utilization{job="nvidia-dcgm-exporter"})