User-Community Airflow Helm Chart完全指南:如何在Kubernetes上部署Apache Airflow
User-Community Airflow Helm Chart完全指南:如何在Kubernetes上部署Apache Airflow
【免费下载链接】chartsThe User-Community Airflow Helm Chart is the standard way to deploy Apache Airflow on Kubernetes with Helm. Originally created in 2017, it has since helped thousands of companies create production-ready deployments of Airflow on Kubernetes.项目地址: https://gitcode.com/gh_mirrors/charts27/charts
User-Community Airflow Helm Chart是在Kubernetes上使用Helm部署Apache Airflow的标准方式。自2017年创建以来,它已帮助数千家公司在Kubernetes上创建了生产就绪的Airflow部署。本指南将为你提供使用该Helm Chart在Kubernetes上部署Apache Airflow的完整步骤,从环境准备到最终访问Airflow UI,让你轻松上手。
准备你的部署环境
在开始部署之前,确保你的环境满足以下要求:
- Kubernetes
1.18+ - Helm
3.0+(installing helm) - (可选) 配置包含DAG文件的Git仓库 (loading dag definitions)
- (可选) 用于
CeleryExecutor的外部PostgreSQL或MySQL数据库 (connecting your database) - (可选) 用于
CeleryExecutor的外部Redis数据库 (connecting your redis)
🟦Tip🟦
要部署
User-Community Airflow Helm Chart,你需要一个Kubernetes集群。以下是一些流行的Kubernetes发行版按平台分类:
Platform Kubernetes Distribution Local Machine k3d Local Machine kind Local Machine minikube Amazon Web Services Amazon Elastic Kubernetes Service (EKS) Microsoft Azure Azure Kubernetes Service (AKS) Google Cloud Google Kubernetes Engine (GKE) Alibaba Cloud Alibaba Cloud Container Service for Kubernetes (ACK) IBM Cloud IBM Cloud Kubernetes Service (IKS)
添加Helm仓库
执行以下命令将仓库添加到你的Helm:
## add this helm repository helm repo add airflow-stable https://airflow-helm.github.io/charts ## update your helm repo cache helm repo update创建自定义Values文件
Helm图表通过values进行配置,图表的所有values列表都在图表的values.yaml文件中(该文件也设置了默认值)。User-Community Airflow Helm Chart有数量极多的values(超过1000个!),但不必担心,你可以从定义一些重要的values开始,然后逐步扩展你的custom-values.yaml。
我们建议你从我们的示例中开始创建custom-values.yaml文件:
CeleryExecutorKubernetesExecutorCeleryKubernetesExecutor
🟦Tip🟦
以下链接将帮助你扩展
custom-values.yaml以满足你的需求:
Docs: Key FeaturesDocs: Frequently Asked QuestionsDocs: ExamplesDocs: Helm Values
安装Airflow Chart
## set the release-name & namespace export AIRFLOW_NAME="airflow-cluster" export AIRFLOW_NAMESPACE="airflow-cluster" ## create the namespace kubectl create ns "$AIRFLOW_NAMESPACE" ## install using helm 3 helm install \ "$AIRFLOW_NAME" \ airflow-stable/airflow \ --namespace "$AIRFLOW_NAMESPACE" \ --version "8.X.X" \ --values ./custom-values.yaml ## wait until the above command returns and resources become ready ## (may take a while)🟥Warning🟥
始终固定
--version,这样你就不会意外更新图表版本!
🟦Tip🟦
- 在我们的CHANGELOG中找到完整的图表版本列表
- 在GitHub上
Watch 👀以获取有关新图表版本的通知,点击"watch"→"custom"→"releases"。
访问Airflow UI
## port-forward the airflow webserver kubectl port-forward svc/${AIRFLOW_NAME}-web 8080:8080 --namespace $AIRFLOW_NAMESPACE ## open your browser to: http://localhost:8080 ## (default login: `admin`/`admin`)🟦Tip🟦
了解更多关于身份验证的信息:
Manage Airflow UsersIntegrate Airflow with LDAP or OAUTH
常见配置与管理
配置数据库
Configure Database (External)Configure Database (Built-In)Configure PgBouncer
配置Redis
Configure Redis (Built-In)Configure Database (External)
管理Airflow组件
Manage Airflow PluginsInstall Extra Python/Pip PackagesConfigure Celery Worker Autoscaling
管理DAGs
Load Airflow DAGsManage Airflow ConnectionsManage Airflow VariablesManage Airflow Pools
安全设置
Set Airflow Fernet Encryption KeySet Airflow Webserver Secret Key
监控与日志
Integrate Airflow with PrometheusManage Airflow LogsManage Airflow Logs Cleanup
Kubernetes相关配置
Configure Kubernetes IngressMount Extra Persistent VolumesMount Files from Secrets/ConfigMapsMount Environment Variables from Secrets/ConfigMapsConfigure Pod Affinity/Selectors/TolerationsInclude Extra Kubernetes Manifests
升级与卸载指南
升级Airflow Chart
如需升级已部署的Airflow Chart,可参考Upgrade Guide。
卸载Airflow Chart
若要卸载Airflow Chart,可按照Uninstall Guide进行操作。
通过本指南,你已经掌握了使用User-Community Airflow Helm Chart在Kubernetes上部署、配置和管理Apache Airflow的基本方法。如有更多需求,可查阅项目的官方文档获取更详细的信息。
【免费下载链接】chartsThe User-Community Airflow Helm Chart is the standard way to deploy Apache Airflow on Kubernetes with Helm. Originally created in 2017, it has since helped thousands of companies create production-ready deployments of Airflow on Kubernetes.项目地址: https://gitcode.com/gh_mirrors/charts27/charts
创作声明:本文部分内容由AI辅助生成(AIGC),仅供参考
