Kubernetes CRD 开发实践指南
Kubernetes CRD 开发实践指南
引言
自定义资源定义(CRD)是 Kubernetes 扩展能力的核心机制,允许用户定义自己的资源类型。本文将深入探讨 CRD 的开发流程、最佳实践和常见模式。
一、CRD 架构概述
1.1 CRD 层次结构
┌─────────────────────────────────────────────────────────────┐ │ CRD 架构 │ ├─────────────────────────────────────────────────────────────┤ │ │ │ ┌─────────────────────────────────────────────────────┐ │ │ │ API Server │ │ │ │ - CRD 注册与验证 │ │ │ │ - REST API 端点生成 │ │ │ └─────────────────────────────────────────────────────┘ │ │ │ │ │ ▼ │ │ ┌─────────────────────────────────────────────────────┐ │ │ │ CRD 资源 │ │ │ │ - CustomResourceDefinition │ │ │ │ - CustomResource │ │ │ └─────────────────────────────────────────────────────┘ │ │ │ │ │ ▼ │ │ ┌─────────────────────────────────────────────────────┐ │ │ │ Controller │ │ │ │ - Reconcile Loop │ │ │ │ - 状态同步 │ │ │ └─────────────────────────────────────────────────────┘ │ │ │ │ │ ▼ │ │ ┌─────────────────────────────────────────────────────┐ │ │ │ 底层资源 │ │ │ │ - Deployment/Pod/Service │ │ │ └─────────────────────────────────────────────────────┘ │ │ │ └─────────────────────────────────────────────────────────────┘1.2 CRD 定义结构
| 组件 | 描述 | 作用 |
|---|---|---|
| Group | API 组名 | 组织相关资源 |
| Version | API 版本 | 版本管理 |
| Scope | 作用域 | Namespaced/Cluster |
| Names | 命名规则 | Kind/ListKind/Plural/Singular |
| Schema | 字段定义 | 验证和默认值 |
二、CRD 定义
2.1 基础 CRD 定义
apiVersion: crd.k8s.io/v1 kind: CustomResourceDefinition metadata: name: databases.example.com spec: group: example.com versions: - name: v1 served: true storage: true schema: openAPIV3Schema: type: object properties: spec: type: object properties: version: type: string replicas: type: integer minimum: 1 maximum: 10 storage: type: string backup: type: object properties: enabled: type: boolean schedule: type: string required: - version - replicas scope: Namespaced names: plural: databases singular: database kind: Database shortNames: - db2.2 CRD 版本管理
apiVersion: crd.k8s.io/v1 kind: CustomResourceDefinition metadata: name: databases.example.com spec: group: example.com versions: - name: v1alpha1 served: true storage: false schema: openAPIV3Schema: type: object properties: spec: type: object properties: version: type: string - name: v1beta1 served: true storage: false schema: openAPIV3Schema: type: object properties: spec: type: object properties: version: type: string replicas: type: integer - name: v1 served: true storage: true schema: openAPIV3Schema: type: object properties: spec: type: object properties: version: type: string replicas: type: integer storage: type: string2.3 CRD 子资源配置
apiVersion: crd.k8s.io/v1 kind: CustomResourceDefinition metadata: name: databases.example.com spec: group: example.com versions: - name: v1 served: true storage: true subresources: status: {} scale: specReplicasPath: .spec.replicas statusReplicasPath: .status.replicas labelSelectorPath: .status.selector schema: openAPIV3Schema: type: object properties: spec: type: object properties: replicas: type: integer status: type: object properties: replicas: type: integer nullable: true selector: type: string nullable: true三、自定义资源使用
3.1 创建自定义资源
apiVersion: example.com/v1 kind: Database metadata: name: my-postgres namespace: default spec: version: "14.0" replicas: 3 storage: "100Gi" backup: enabled: true schedule: "0 2 * * *"3.2 查看自定义资源
# 查看所有 Database 资源 kubectl get databases # 查看特定资源 kubectl get database my-postgres # 查看详细信息 kubectl describe database my-postgres # 输出 YAML 格式 kubectl get database my-postgres -o yaml四、Controller 开发
4.1 Controller 结构
package main import ( "context" "time" "github.com/go-logr/logr" "k8s.io/apimachinery/pkg/runtime" "sigs.k8s.io/controller-runtime/pkg/client" "sigs.k8s.io/controller-runtime/pkg/controller" "sigs.k8s.io/controller-runtime/pkg/handler" "sigs.k8s.io/controller-runtime/pkg/log" "sigs.k8s.io/controller-runtime/pkg/manager" "sigs.k8s.io/controller-runtime/pkg/reconcile" "sigs.k8s.io/controller-runtime/pkg/source" examplev1 "example.com/api/v1" ) type DatabaseReconciler struct { client.Client Scheme *runtime.Scheme Log logr.Logger } func (r *DatabaseReconciler) Reconcile(ctx context.Context, req reconcile.Request) (reconcile.Result, error) { log := log.FromContext(ctx) var db examplev1.Database if err := r.Get(ctx, req.NamespacedName, &db); err != nil { return reconcile.Result{}, client.IgnoreNotFound(err) } log.Info("Reconciling Database", "name", db.Name) // 业务逻辑处理 if err := r.reconcileDeployment(ctx, &db); err != nil { return reconcile.Result{}, err } if err := r.reconcileService(ctx, &db); err != nil { return reconcile.Result{}, err } if err := r.updateStatus(ctx, &db); err != nil { return reconcile.Result{}, err } return reconcile.Result{RequeueAfter: time.Minute * 5}, nil } func main() { mgr, err := manager.New(cfg, manager.Options{}) if err != nil { log.Error(err, "Unable to start manager") os.Exit(1) } if err := (&DatabaseReconciler{ Client: mgr.GetClient(), Scheme: mgr.GetScheme(), Log: log.Log.WithName("controllers").WithName("Database"), }).SetupWithManager(mgr); err != nil { log.Error(err, "Unable to create controller", "controller", "Database") os.Exit(1) } if err := mgr.Start(ctrl.SetupSignalHandler()); err != nil { log.Error(err, "Manager exited non-zero") os.Exit(1) } }4.2 Reconcile 逻辑
func (r *DatabaseReconciler) reconcileDeployment(ctx context.Context, db *examplev1.Database) error { // 检查 Deployment 是否存在 var deploy appsv1.Deployment deployName := db.Name if err := r.Get(ctx, client.ObjectKey{Namespace: db.Namespace, Name: deployName}, &deploy); err != nil { if client.IgnoreNotFound(err) != nil { return err } // 创建 Deployment deploy = *newDeployment(db) if err := r.Create(ctx, &deploy); err != nil { return err } return nil } // 更新 Deployment if !reflect.DeepEqual(deploy.Spec.Replicas, &db.Spec.Replicas) { deploy.Spec.Replicas = &db.Spec.Replicas if err := r.Update(ctx, &deploy); err != nil { return err } } return nil }五、CRD 最佳实践
5.1 版本化策略
versions: - name: v1alpha1 served: true storage: false - name: v1beta1 served: true storage: false - name: v1 served: true storage: true5.2 状态管理
subresources: status: {}func (r *DatabaseReconciler) updateStatus(ctx context.Context, db *examplev1.Database) error { db.Status.ReadyReplicas = 3 db.Status.Phase = "Ready" if err := r.Status().Update(ctx, db); err != nil { return err } return nil }5.3 验证与默认值
schema: openAPIV3Schema: type: object properties: spec: type: object properties: replicas: type: integer minimum: 1 maximum: 10 default: 1 version: type: string enum: - "12.0" - "13.0" - "14.0" default: "14.0"六、CRD 部署
6.1 RBAC 配置
apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRole metadata: name: database-manager rules: - apiGroups: ["example.com"] resources: ["databases", "databases/status"] verbs: ["get", "list", "watch", "create", "update", "patch", "delete"] - apiGroups: ["apps"] resources: ["deployments"] verbs: ["get", "list", "watch", "create", "update", "patch", "delete"] - apiGroups: [""] resources: ["services"] verbs: ["get", "list", "watch", "create", "update", "patch", "delete"] --- apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRoleBinding metadata: name: database-manager roleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole name: database-manager subjects: - kind: ServiceAccount name: database-controller namespace: default6.2 Controller 部署
apiVersion: apps/v1 kind: Deployment metadata: name: database-controller spec: replicas: 1 selector: matchLabels: app: database-controller template: metadata: labels: app: database-controller spec: serviceAccountName: database-controller containers: - name: controller image: my-registry/database-controller:latest args: - --metrics-addr=:8080 - --enable-leader-election resources: limits: cpu: 100m memory: 128Mi requests: cpu: 100m memory: 128Mi七、常见问题与解决方案
7.1 CRD 创建失败
问题分析:
- Schema 定义错误
- 版本配置错误
- 命名冲突
解决方案:
# 检查 CRD 状态 kubectl get crd databases.example.com -o yaml # 查看事件 kubectl describe crd databases.example.com7.2 Controller 无法获取资源
问题分析:
- RBAC 权限不足
- 资源未正确注册
- 缓存未同步
解决方案:
# 检查 ServiceAccount 权限 kubectl auth can-i get databases --as=system:serviceaccount:default:database-controller7.3 状态更新失败
问题分析:
- 未配置 subresources.status
- 更新策略错误
- 并发冲突
解决方案:
subresources: status: {}// 使用 Status().Update if err := r.Status().Update(ctx, db); err != nil { if apierrors.IsConflict(err) { return reconcile.Result{Requeue: true}, nil } return err }结论
CRD 是 Kubernetes 扩展能力的核心机制,通过定义自定义资源和开发 Controller,可以将复杂业务逻辑封装为 Kubernetes 原生资源。遵循版本化、状态管理、验证等最佳实践,可以构建稳定、可维护的 CRD 解决方案。
