Kubernetes 基于sealos创建k8s集群

实践环境

openEuler-22.03-LTS-SP4

registry.cn-shanghai.aliyuncs.com/labring/kubernetes:v1.27.16

registry.cn-shanghai.aliyuncs.com/labring/helm:v3.8.2

registry.cn-shanghai.aliyuncs.com/labring/cilium:v1.14.4

https://github.com/labring/sealos/releases/download/v5.1.1/sealos_5.1.1_linux_amd64.tar.gz

简介

Sealos是一个简单的 Golang 二进制文件，可用于快速部署Kubernetes集群

• 支持在线和离线安装，适用于amd64和arm64架构的 K8s 集群。

• 支持节点管理和分布式应用安装。

• 支持Containerd和Docker运行时。

• 支持大多数 Linux 发行版，例如：Ubuntu、CentOS、Rocky linux。
• 支持 Docker Hub 中的所有 Kubernetes 版本。
• 支持使用 Containerd 作为容器运行时。

先决条件

以下是一些基本的安装要求：

• 每个集群节点的主机名保持唯一，且主机名不要带下划线。

• 所有节点的时间需要保持一致。

• 建议使用干净的操作系统来创建集群。不要自己装 Docker！

• 主节点内存 尽量大于等于3 G，否则运行时可能因为内存不足报错

  [ERROR Mem]: the system RAM (1427 MB) is less than the minimum 1700 MB
  

• 建议不要创建/var分区，如果主节点有创建/var分区，建议配置50G以上，用于存储相关镜像，其它worker节点如果有创建/var分区，也建议配置大一点20G以上

前置准备

1、同步所有集群节点的时间

2、修改所集群节点的主机名

配置示例--设置节点 192.168.88.141的主机名

# hostnamectl set-hostname 192-168-88-141

3、 关闭防火墙

# systemctl stop firewalld
# systemctl disable firewalld

4、选择k8s集群镜像版本

浏览器打开 Registry Explorer ，可以查看 K8s 集群镜像的所有版本：

输入 registry.cn-shanghai.aliyuncs.com/labring/kubernetes，然后点击“提交”：

就会看到这个集群镜像的所有 tag。

Docker Hub 同理，输入 docker.io/labring/kubernetes 即可查看所有 tag。

注意：K8s 的小版本号越高，集群越稳定。例如 v1.29.x，其中的 x 就是小版本号。建议使用小版本号比较高的 K8s 版本。本文截止前，v1.27 最高的版本号是 v1.27.16，而 v1.31 最高的版本号是 v1.31.9，所以建议使用 v1.27.16。你需要根据实际情况来选择最佳的 K8s 版本

5、明确适配所选k8s版本的 labring/helm 和labring/cilium镜像版本

6、下载 Sealos并配置

手动下载地址：https://github.com/labring/sealos/releases

注意

1、Sealos的版本需要适配k8s集群镜像版本，详情参见：集群镜像版本支持说明

2、建议使用稳定版本例如v4.3.0。像 v4.3.0-rc1、v4.3.0-alpha1 这样的版本是预发布版，请谨慎使用。

3、master节点执行

这里选择下载二进制

# wget https://github.com/labring/sealos/releases/download/v5.1.1/sealos_5.1.1_linux_amd64.tar.gz && tar -zxvf sealos_5.1.1_linux_amd64.tar.gz sealos && chmod +x sealos && mv sealos /usr/bin/

说明：如果无法直接下载(比如在内网，无法直接访问网络)，可以外网下载然后再上传服务器执行解压等操作。

参考连接：https://sealos.run/docs/k8s/quick-start/install-cli

4、master执行

yum install -y socat

解决安装过程中出现告警：[WARNING FileExisting-socat]: socat not found in system path

5、

安装K8S集群

方式1、在线安装

master节点上执行

# sealos run registry.cn-shanghai.aliyuncs.com/labring/kubernetes:v1.22.17 registry.cn-shanghai.aliyuncs.com/labring/helm:v3.8.2 registry.cn-shanghai.aliyuncs.com/labring/cilium:v1.14.4 \--masters 192.168.88.139 \--nodes 192.168.88.140,192.168.88.141 -p testpwd@316

注意：labring/helm 应当在 labring/cilium 之前。

参数说明：

• --masters IP列表 K8s master 节点地址列表，如果有多个master节点即多个IP地址，IP之间用英文逗号分隔，形如 192.168.64.2,192.168.64.2
• --nodes IP列表 K8s node 节点地址列表，地址之间用英文逗号分隔
• -p 节点ssh登录密码

遇到问题

实际安装过程中，遇到过安装失败的情况，错误提示如下：

[wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s
[kubelet-check] Initial timeout of 40s passed.Unfortunately, an error has occurred:timed out waiting for the conditionThis error is likely caused by:- The kubelet is not running- The kubelet is unhealthy due to a misconfiguration of the node in some way (required cgroups disabled)If you are on a systemd-powered system, you can try to troubleshoot the error with the following commands:- 'systemctl status kubelet'- 'journalctl -xeu kubelet'Additionally, a control plane component may have crashed or exited when started by the container runtime.To troubleshoot, list all containers using your preferred container runtimes CLI.Here is one example how you may list all Kubernetes containers running in cri-o/containerd using crictl:- 'crictl --runtime-endpoint unix:///run/containerd/containerd.sock ps -a | grep kube | grep -v pause'Once you have found the failing container, you can inspect its logs with:- 'crictl --runtime-endpoint unix:///run/containerd/containerd.sock logs CONTAINERID'error execution phase wait-control-plane: couldn't initialize a Kubernetes cluster
To see the stack trace of this error execute with --v=5 or higher
2026-01-17T02:32:51 error Applied to cluster error: failed to init masters: init master0 failed, error: exit status 1. Please clean and reinstall
Error: failed to init masters: init master0 failed, error: exit status 1. Please clean and reinstall

查看kubelet 状态如下

# systemctl status kubelet
● kubelet.service - kubelet: The Kubernetes Node AgentLoaded: loaded (/etc/systemd/system/kubelet.service; enabled; vendor preset: disabled)Drop-In: /etc/systemd/system/kubelet.service.d└─10-kubeadm.confActive: active (running) since Sat 2026-01-17 02:17:08 CST; 5min agoDocs: http://kubernetes.io/docs/Process: 2221 ExecStartPre=/usr/bin/kubelet-pre-start.sh (code=exited, status=0/SUCCESS)Main PID: 2237 (kubelet)Tasks: 13 (limit: 15376)Memory: 42.7MCGroup: /system.slice/kubelet.service└─ 2237 /usr/bin/kubelet --bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/etc/kubernetes/kubelet.conf --config=/var/lib/kubelet/config.yaml --container-runti>Jan 17 02:22:32 192-168-88-139 kubelet[2237]: E0117 02:22:32.852248    2237 kubelet.go:2456] "Error getting node" err="node \"192-168-88-139\" not found"
.....

查看kubelet系统日志，发现存在以下类似以下错误

Jan 17 11:32:37 192-168-88-139 kubelet[280750]: I0117 11:32:37.489925  280750 dynamic_cafile_content.go:155] "Starting controller" name="client-ca-bundle::/etc/kubernetes/pki/ca.crt"
Jan 17 11:32:37 192-168-88-139 kubelet[280750]: E0117 11:32:37.495435  280750 certificate_manager.go:471] kubernetes.io/kube-apiserver-client-kubelet: Failed while requesting a signed certificate from the control plane: cannot create certificate signing request: Post "https://apiserver.cluster.local:6443/apis/certificates.k8s.io/v1/certificatesigningrequests": dial tcp 192.168.88.139:6443: connect: connection refused

Jan 17 11:33:51 192-168-88-139 kubelet[280750]: E0117 11:33:51.604218  280750 pod_workers.go:951] "Error syncing pod, skipping" err="failed to \"CreatePodSandbox\" for \"kube-apiserver-192-168-88-139_kube-system(7eb23211a94fd3a4a50291a818fefe89)\" with CreatePodSandboxError: \"Failed to create sandbox for pod \\\"kube-apiserver-192-168-88-139_kube-system(7eb23211a94fd3a4a50291a818fefe89)\\\": rpc error: code = Unknown desc = failed to create containerd task: failed to create shim task: OCI runtime create failed: unable to retrieve OCI runtime error (open /run/containerd/io.containerd.runtime.v2.task/k8s.io/23a20fe1613712213d8ff67507c9c81639cc6d75d63b6728682df689a0f9a970/log.json: no such file or directory): fork/exec /usr/bin/runc: exec format error: unknown\"" pod="kube-system/kube-apiserver-192-168-88-139" podUID=7eb23211a94fd3a4a50291a818fefe89

说明：当然除了上述错误日志还有其它非关键错误日志，笔者排查后选择性忽略了。

查看文件

# file /usr/bin/runc
/usr/bin/runc: ASCII text, with no line terminators

初步断定 /usr/bin/runc 文件生成失败，导致kubelet无法正常运行，从而导致主节点注册失败

解决方法

先执行以下命令，清理k8s集群，然后重新运行上述安装命令，

# sealos reset

安装过程中(出现: fork/exec /usr/bin/runc: exec format error: unknown错误时)，手动下载runc文件并替换

# wget https://github.com/opencontainers/runc/releases/download/v1.1.12/runc.amd64 -O /usr/bin/runc

问题：为啥不是在运行sealos前替换呢？因为sealos会动态创建该文件，运行前替换会被覆盖。

集群安装好后，查看集群节点状态，如下发现存在非就绪状态节点

# kubectl get nodes
NAME             STATUS     ROLES                  AGE   VERSION
192-168-88-139   Ready      control-plane,master   27m   v1.22.17
192-168-88-140   NotReady   <none>                 27m   v1.22.17
192-168-88-141   NotReady   <none>                 27m   v1.22.17

查看 kubelet 系统日志，发现存在以下关键错误日志

Jan 17 12:41:32 192-168-88-139 kubelet[302969]: E0117 12:41:32.954702  302969 kuberuntime_manager.go:819] "CreatePodSandbox for pod failed" err="rpc error: code = Unknown desc = failed to setup network for sandbox \"2f60f86855778cfab8037eb27d657ff3254bc58c1ecfc206dd984cfe41978f43\": plugin type=\"cilium-cni\" failed (add): unable to connect to Cilium daemon: failed to create cilium agent client after 30.000000 seconds timeout: Get \"http://localhost/v1/config\": dial unix /var/run/cilium/cilium.sock: connect: no such file or directory\nIs the agent running?" pod="kube-system/coredns-7bdbbf6bf5-99cf4"
Jan 17 12:41:32 192-168-88-139 kubelet[302969]: E0117 12:41:32.954738  302969 pod_workers.go:951] "Error syncing pod, skipping" err="failed to \"CreatePodSandbox\" for \"coredns-7bdbbf6bf5-99cf4_kube-system(7f589667-5cac-4c4c-b993-459318dfb8bd)\" with CreatePodSandboxError: \"Failed to create sandbox for pod \\\"coredns-7bdbbf6bf5-99cf4_kube-system(7f589667-5cac-4c4c-b993-459318dfb8bd)\\\": rpc error: code = Unknown desc = failed to setup network for sandbox \\\"2f60f86855778cfab8037eb27d657ff3254bc58c1ecfc206dd984cfe41978f43\\\": plugin type=\\\"cilium-cni\\\" failed (add): unable to connect to Cilium daemon: failed to create cilium agent client after 30.000000 seconds timeout: Get \\\"http://localhost/v1/config\\\": dial unix /var/run/cilium/cilium.sock: connect: no such file or directory\\nIs the agent running?\"" pod="kube-system/coredns-7bdbbf6bf5-99cf4" podUID=7f589667-5cac-4c4c-b993-459318dfb8bd
Jan 17 12:41:34 192-168-88-139 kubelet[302969]: E0117 12:41:34.758111  302969 cadvisor_stats_provider.go:415] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/system.slice/kubelet.service\": RecentStats: unable to find data in memory cache]"

根据日志分析可知cilium运行状态异常，导致节点资源监控（cAdvisor）数据收集受阻，属于连带问题， 查看其pod状态，发现全部异常

# kubectl get pods -n kube-system | grep cilium
cilium-2s77x                             0/1     Init:0/6            0               34m
cilium-operator-6778f57859-ls6qn         0/1     ContainerCreating   0               34m
cilium-rqr6f                             0/1     Running             8 (6m42s ago)   34m
cilium-wbjf7                             0/1     Init:0/6            0               34m

查看pod事件，发现以下错误

Events:Type     Reason                  Age                    From               Message----     ------                  ----                   ----               -------Normal   Scheduled               34m                    default-scheduler  Successfully assigned kube-system/cilium-2s77x to 192-168-88-141Warning  FailedCreatePodSandBox  34m                    kubelet            Failed to create pod sandbox: rpc error: code = Unknown desc = failed to create containerd task: failed to create shim task: OCI runtime create failed: unable to retrieve OCI runtime error (open /run/containerd/io.containerd.runtime.v2.task/k8s.io/655252912fe2c7336eb25a1d57e78a7bcaff2fb7a98890c11000454dd10d2b7b/log.json: no such file or directory): fork/exec /usr/bin/runc: exec format error: unknown

解决方法：每个节点上执行以下命令，手动替换runc 二进制

# wget https://github.com/opencontainers/runc/releases/download/v1.1.12/runc.amd64 -O /usr/bin/runc

然后重启cilium daemonset

# kubectl rollout restart daemonset cilium -n kube-system

再次检测节点状态，如下，都正常。至此集群部署成功。

#  kubectl get nodes
NAME             STATUS   ROLES                  AGE   VERSION
192-168-88-139   Ready    control-plane,master   55m   v1.22.17
192-168-88-140   Ready    <none>                 54m   v1.22.17
192-168-88-141   Ready    <none>                 54m   v1.22.17

方式2：离线安装

离线环境只需要提前导入镜像，其它步骤与在线安装一致。

kubernetes为例，首先在有网络的环境中导出集群镜像：

# sealos pull registry.cn-shanghai.aliyuncs.com/labring/kubernetes:v1.22.17
# sealos save -o kubernetes.tar registry.cn-shanghai.aliyuncs.com/labring/kubernetes:v1.22.17

导入镜像并安装，将 kubernetes.tar 拷贝到离线环境，使用 load 命令导入镜像即可：

# sealos load -i kubernetes.tar
sealos images # 查看集群镜像是否导入成功

剩下的安装方式与在线安装的步骤一致：

# run registry.cn-shanghai.aliyuncs.com/labring/kubernetes:v1.22.17 registry.cn-shanghai.aliyuncs.com/labring/helm:v3.8.2 registry.cn-shanghai.aliyuncs.com/labring/cilium:v1.14.4 \--masters 192.168.88.139 \--nodes 192.168.88.140,192.168.88.141 -p testpwd@316

也可以不用 load 命令导入镜像，直接运行以下启动命令即可安装 K8s：

# sealos run kubernetes.tar helm.tar cilium.tar--masters 192.168.88.139 \--nodes 192.168.88.140,192.168.88.141 -p testpwd@316

按需安装其它分布式应用

示例：

sealos run registry.cn-shanghai.aliyuncs.com/labring/openebs:v3.9.0 # install openebs
sealos run registry.cn-shanghai.aliyuncs.com/labring/minio-operator:v4.5.5 registry.cn-shanghai.aliyuncs.com/labring/ingress-nginx:4.1.0

这样Minio，openebs 等应用都有了，不用关心所有的依赖问题。

附：sealos其它功能命令简介

增加 K8s 节点

增加 node 节点：

$ sealos add --nodes 192.168.88.142,192.168.88.143

增加 master 节点：

$ sealos add --masters 192.168.88.137,192.168.88.138

删除 K8s 节点

删除 node 节点：

$ sealos delete --nodes 192.168.88.142,192.168.88.143

删除 master 节点：

$ sealos delete --masters 192.168.88.137,192.168.88.138

清理 K8s 集群

$ sealos reset

更多用法，查看命令帮助 sealos --help

参考链接

https://sealos.run/docs/k8s/quick-start/deploy-kubernetes