2.6.升级 containerd
更新前/usr/local/bin/下的 containerd 相关文件:
# ll /usr/local/bin/ total 279M lrwxrwxrwx 1 root root 33 Jul 14 2025 nerdctl -> /usr/local/containerd/bin/nerdctl -rwxr-xr-x 1 demo demo 23M Mar 28 2025 etcd -rwxr-xr-x 1 demo demo 18M Mar 28 2025 etcdctl -rwxr-xr-x 1 root root 27M Mar 18 2025 ctd-decoder -rwxr-xr-x 1 root root 55M Mar 18 2025 crictl -rwxr-xr-x 1 root root 56M Mar 18 2025 critest -rwxr-xr-x 1 root root 39M Mar 18 2025 containerd -rwxr-xr-x 1 root root 6.4M Mar 18 2025 containerd-shim -rwxr-xr-x 1 root root 7.4M Mar 18 2025 containerd-shim-runc-v1 -rwxr-xr-x 1 root root 13M Mar 18 2025 containerd-shim-runc-v2 -rwxr-xr-x 1 root root 18M Mar 18 2025 containerd-stress -rwxr-xr-x 1 root root 19M Mar 18 2025 ctr解压新版本(--strip-components=1去掉压缩包内的bin/目录层级,直接覆盖到/usr/local/bin/):
# tar tf containerd-2.3.2-linux-amd64.tar.gz bin/ bin/containerd bin/containerd-shim-runc-v2 bin/ctr bin/containerd-stress # tar -xzvf containerd-2.3.2-linux-amd64.tar.gz --strip-components=1 -C /usr/local/bin/更新后(containerd 主程序及相关组件更新为Jun 19时间戳;注意旧版的containerd-shim、containerd-shim-runc-v1等 2.0 已废弃的 shim 仍保留,但不再使用):
# ll /usr/local/bin/ total 286M -rwxr-xr-x 1 root root 42M Jun 19 07:14 containerd -rwxr-xr-x 1 root root 8.1M Jun 19 07:14 containerd-shim-runc-v2 -rwxr-xr-x 1 root root 22M Jun 19 07:14 containerd-stress -rwxr-xr-x 1 root root 25M Jun 19 07:14 ctr lrwxrwxrwx 1 root root 33 Jul 14 2025 nerdctl -> /usr/local/containerd/bin/nerdctl -rwxr-xr-x 1 demo demo 23M Mar 28 2025 etcd -rwxr-xr-x 1 demo demo 18M Mar 28 2025 etcdctl -rwxr-xr-x 1 root root 27M Mar 18 2025 ctd-decoder -rwxr-xr-x 1 root root 55M Mar 18 2025 crictl -rwxr-xr-x 1 root root 56M Mar 18 2025 critest # 2.0 已废弃 -rwxr-xr-x 1 root root 6.4M Mar 18 2025 containerd-shim # 2.0 已废弃 -rwxr-xr-x 1 root root 7.4M Mar 18 2025 containerd-shim-runc-v12.7.重启 containerd 并验证
systemctl restart containerd systemctl status containerd3.验证升级结果
逐节点升级完成后,确认运行时版本已更新为containerd://2.3.2:
# kubectl get node -o wide NAME STATUS ROLES AGE VERSION INTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME x.x.x Ready control-plane 344d v1.35.6 x.x.x.x Ubuntu 24.04.4 LTS 6.8.0-124-generic containerd://2.3.2 x.x.x Ready control-plane 344d v1.35.6 x.x.x.x Ubuntu 24.04.4 LTS 6.8.0-124-generic containerd://2.3.2 x.x.x Ready control-plane 344d v1.35.6 x.x.x.x Ubuntu 24.04.4 LTS 6.8.0-124-generic containerd://2.3.2问题处理
1.Kubernetes 问题处理
1.1.基础镜像版本不一致
问题原因
新版 kubeadm 默认使用的 pause 镜像版本与节点上 containerd 配置的不一致。
表现形式
在执行 kubeadm upgrade apply 时,可能会出现基础镜像不一致的告警
detected that the sandbox image "registry.cn-hangzhou.aliyuncs.com/google_containers/pause:3.8" of the container runtime is inconsistent with that used by kubeadm. It is recommended to use "registry.cn-hangzhou.aliyuncs.com/google_containers/pause:3.10" as the CRI sandbox image.解决方式
查看当前 kubeadm 期望的镜像版本:
# kubeadm config images list # 此处报错是由于无法连接 "https://dl.k8s.io/release/stable-1.txt",但不影响输出所需镜像 W0626 10:30:12.306990 2755682 version.go:108] could not fetch a Kubernetes version from the internet: unable to get URL "https://dl.k8s.io/release/stable-1.txt": Get "https://dl.k8s.io/release/stable-1.txt": context deadline exceeded (Client.Timeout exceeded while awaiting headers) W0626 10:30:12.307154 2755682 version.go:109] falling back to the local client version: v1.35.6 registry.k8s.io/kube-apiserver:v1.35.6 registry.k8s.io/kube-controller-manager:v1.35.6 registry.k8s.io/kube-scheduler:v1.35.6 registry.k8s.io/kube-proxy:v1.35.6 registry.k8s.io/coredns/coredns:v1.13.1 registry.k8s.io/pause:3.10.1 registry.k8s.io/etcd:3.6.6-0 # 查看当前 containerd 实际配置 grep -i 'pause:' /etc/containerd/config.toml sandbox_image = "registry.cn-hangzhou.aliyuncs.com/google_containers/pause:3.8"把 containerd 配置中的 pause 版本更新到提示的新版本,然后重启 containerd 即可:
sed -i 's#pause:3.8#pause:3.10.1#' /etc/containerd/config.toml systemctl restart containerd1.2.集群更新使用的版本格式问题
问题原因
升级时kubeadm upgrade apply所用版本不是 K8s 版本号
表现形式
升级时提示版本无效
# kubeadm upgrade apply 1.33.13-1.1 error execution phase preflight: the version argument is invalid due to these errors: - Specified version to upgrade to "v1.33.13-1.1" is an unstable version and such upgrades weren't allowed via setting the --allow-*-upgrades flags Can be bypassed if you pass the --force flag解决方式
kubeadm upgrade apply所用的版本必须是K8s 版本号(如1.33.13),而不是 apt 包的 Ubuntu 格式(如1.33.13-1.1):
# kubeadm upgrade apply 1.33.131.3.集群配置中 controlPlaneEndpoint 与实际 bindPort 不一致
问题原因
这个告警无需在意,它只是提示controlPlaneEndpoint覆盖了bindPort。本集群用 HAProxy 做负载均衡,前端监听16443,再转发给后端 kube-apiserver 的6443,所以这个覆盖正是预期行为。
表现形式
升级过程中 kubeadm upgrade 输出:
WARNING: port specified in controlPlaneEndpoint overrides bindPort in the controlplane address解决方式
符合预期即可,无需解决:
# ss -nltup | grep 6443 tcp LISTEN 0 2000 0.0.0.0:16443 0.0.0.0:* users:(("haproxy",pid=1892947,fd=10)) tcp LISTEN 0 2000 127.0.0.1:16443 0.0.0.0:* users:(("haproxy",pid=1892947,fd=11)) tcp LISTEN 0 16384 *:6443 *:* users:(("kube-apiserver",pid=1697748,fd=4)) # kubectl -n kube-system get cm kubeadm-config -o yaml | grep -i 'controlPlaneEndpoint' controlPlaneEndpoint: 10.51.0.249:16443 # cat /etc/haproxy/haproxy.cfg global maxconn 2000 ulimit-n 16384 log 127.0.0.1 local0 err stats timeout 30s defaults log global mode http option httplog timeout connect 5000 timeout client 50000 timeout server 50000 timeout http-request 15s timeout http-keep-alive 15s frontend monitor-in bind *:33305 mode http option httplog monitor-uri /monitor frontend k8s-master bind 0.0.0.0:16443 bind 127.0.0.1:16443 mode tcp option tcplog tcp-request inspect-delay 5s default_backend k8s-master backend k8s-master mode tcp option tcplog option tcp-check balance roundrobin default-server inter 10s downinter 5s rise 2 fall 2 slowstart 60s maxconn 250 maxqueue 256 weight 100 server x.x.x x.x.x.x:6443 check server x.x.x x.x.x.x:6443 check server x.x.x x.x.x.x:6443 check1.4.集群更新使用的 containerd 版本过旧
问题原因
使用的 containerd 版本较旧:K8s 1.36 版本后会删除 "从 kubelet 配置读 cgroupDriver" 这个老机制(fallback),改用 CRI 的RuntimeConfig方法从 containerd 直接探测。但当前 containerd 1.7 不支持这个新方法,所以出现警告。根据官网说明,在 K8s 1.38 版本前完成 containerd 升级即可。
表现形式
在v1.34.x升级到v1.35.x时,kubeadm 输出ContainerRuntimeVersion告警:
# kubeadm upgrade apply 1.35.6 [upgrade/preflight] Running preflight checks [WARNING ContainerRuntimeVersion]: You must update your container runtime to a version that supports the CRI method RuntimeConfig. Falling back to using cgroupDriver from kubelet config will be removed in 1.36. For more information, see https://git.k8s.io/enhancements/keps/sig-node/4033-group-driver-detection-over-cri解决方式
升级 containerd 版本即可
