京公网安备11010502063487号kubectl drain 到底做了什么?为什么总要加一堆 flag?PodDisruptionBudget 怎么让它卡住?uncordon 之后 Pod 会回来吗?这篇在真集群里一步步加 flag、再制造 PDB 阻塞,把输出原样记下来。
kubectl cordon worker:node/kx-main-worker cordoned
节点 spec.unschedulable=True;taints=[('node.kubernetes.io/unschedulable', 'NoSchedule')]
kubectl get nodes:kx-main-control-plane Ready control-plane 9h v1.37 | kx-main-worker Ready,SchedulingDisabled <none> 9h | kx-main-worker2 Ready <none> 9h v1.37.0
已有的 Pod 不受影响:{'worker': ['bare', 'dep', 'dep', 'dep', 'ds-dfndg', 'scratch'], 'worker2': ['ds-9dnbz']}
扩到 5 个副本:新增的 2 个去了哪:{'worker': ['bare', 'dep', 'dep', 'dep', 'ds-dfndg', 'scratch'], 'worker2': ['dep', 'dep', 'ds-9dnbz']}(只能去 worker2)kubectl cordon 把节点的 spec.unschedulable 设为 true,同时带上污点 node.kubernetes.io/unschedulable:NoSchedule,kubectl get nodes 显示 Ready,SchedulingDisabled。节点上已有的 Pod 不动;把 Deployment 扩到 5 个副本,新增的 2 个只能去 worker2。
直接 kubectl drain worker:退出码 1;error: unable to drain node "kx-main-worker" due to error: [cannot delete Pods that declare no controller (use --force to override): k77/bare, cannot delete DaemonSet-managed Pods (use --ignore-daemonsets to ignore): k77/ds-dfndg, kube-system/kindnet-bhd99, kube-system/kube-proxy-btrjt, cannot delete Pods with local storage (use --delete-emptydir-data to override): k77/scratch, kube-system/metrics-server-59b55d8c77-b6q9g], continuing command...
加 --ignore-daemonsets:退出码 1;error: unable to drain node "kx-main-worker" due to error: [cannot delete Pods that declare no controller (use --force to override): k77/bare, cannot delete Pods with local storage (use --delete-emptydir-data to override): k77/scratch, kube-system/metrics-server-59b55d8c77-b6q9g], continuing command...
再加 --delete-emptydir-data:退出码 1;error: unable to drain node "kx-main-worker" due to error: cannot delete Pods that declare no controller (use --force to override): k77/bare, k77/scratch, continuing command...
再加 --force:退出码 0,用时 3.1 秒;输出:
node/kx-main-worker already cordoned
evicting pod kube-system/metrics-server-59b55d8c77-b6q9g
evicting pod k77/scratch
evicting pod k77/bare
evicting pod k77/dep-5b88b8485b-lqfds
pod/metrics-server-59b55d8c77-b6q9g evicted
pod/bare evicted
pod/scratch evicted
pod/dep-5b88b8485b-lqfds evicted
node/kx-main-worker drained
Warning: deleting Pods that declare no controller: k77/bare, k77/scratch; ignoring DaemonSet-managed Pods: k77/ds-dfndg, kube-system/kindnet-bhd99, ku
drain 之后各节点上的 Pod:{'worker2': ['dep', 'dep', 'dep', 'ds-9dnbz'], 'worker': ['ds-dfndg']}
bare(裸 Pod,--force 之后被直接删掉)还在吗:False;scratch(带 emptyDir 的裸 Pod)还在吗:False;dep 的 3 个副本都在 worker2:3 个;DaemonSet 在 worker 上的 Pod 还在:True直接 kubectl drain 报错,列出三类不能删的:没有控制器的 bare、DaemonSet 管理的(ds、kindnet、kube-proxy)、带本地存储的(scratch、metrics-server)。加 --ignore-daemonsets 后剩两类;再加 --delete-emptydir-data 后只剩没有控制器的 bare 和 scratch;加 --force 才成功,3.1 秒内驱逐了 metrics-server、scratch、bare 和一个 dep Pod。结果是 dep 的 3 个副本都在 worker2,DaemonSet 的 Pod 仍留在 worker 上,bare 和 scratch 被直接删掉,不会重建。
PDB dep-pdb:minAvailable=3;当前 healthy=3;disruptionsAllowed=0
drain worker2(上面有 dep 的 3 个副本):25 秒后退出码 1;输出(去重):
node/kx-main-worker2 cordoned
evicting pod kube-system/metrics-server-59b55d8c77-g5mj7
evicting pod k77/dep-…
pod/metrics-server-59b55d8c77-g5mj7 evicted
Warning: ignoring DaemonSet-managed Pods: k77/ds-9dnbz, kube-system/kindnet-bnhqq, kube-system/kube-proxy-bc8wj
error when evicting pods/"dep-… -n "k77" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
这时 Pod 仍然在 worker2 上的有:['dep-5b88'],节点已被 cordon:True(drain 失败也不会撤销 cordon)
删掉 PDB 后重新 drain:退出码 0,用时 4.1 秒;之后 {'worker': ['dep', 'dep', 'dep', 'ds-dfndg'], 'worker2': ['ds-9dnbz']}PDB dep-pdb(minAvailable: 3,当前 healthy=3)的 disruptionsAllowed 是 0。drain worker2:metrics-server 被驱逐,dep 的 Pod 报 Cannot evict pod as it would violate the pod's disruption budget,drain 每 5 秒重试一次,25 秒超时退出码 1;Pod 仍在 worker2 上,节点却已经被 cordon——drain 失败也不会撤销 cordon。删掉 PDB 后重新 drain,4.1 秒完成。
uncordon 两个节点后:{'worker': ['dep', 'dep', 'dep', 'ds-dfndg'], 'worker2': ['ds-9dnbz']}(Pod 不会自己迁回;需要重建/rollout 才会重新分布)
worker2 的 spec.unschedulable=None;taints=Nonekubectl uncordon 只清除 spec.unschedulable 和污点,Pod 位置不变:worker 上还是 dep ×3 和 ds,worker2 上只有 ds,不会自己迁回。需要均衡就 rollout restart 或使用 descheduler。