京公网安备11010502063487号Pod 一直 Pending,0/5 nodes are available 后面那一串到底在说什么?nodeSelector 写错了会怎样?nodeName 是怎么回事?这篇用故意调度不上的 Pod 把 scheduler 的各类淘汰原因都打出来,并验证标签后加、标签后删、nodeName 绕过调度器这几件事。
worker 的标签(5 个):
beta.kubernetes.io/arch=arm64
beta.kubernetes.io/os=linux
kubernetes.io/arch=arm64
kubernetes.io/hostname=kx-sched-worker
kubernetes.io/os=linux
控制面比 worker 多的标签:['node-role.kubernetes.io/control-plane', 'node.kubernetes.io/exclude-from-external-load-balancers']worker 节点只有 5 个标签:kubernetes.io/hostname、kubernetes.io/arch、kubernetes.io/os 以及两个 beta.kubernetes.io/… 的老名字;控制面多 node-role.kubernetes.io/control-plane 和 exclude-from-external-load-balancers。其他的(磁盘类型、GPU、可用区)都得自己 kubectl label 或由云厂商、插件打上。
on-ssd nodeSelector disk=ssd(worker2、worker3 有):worker3 on-ssd-gpu nodeSelector disk=ssd 且 gpu=yes(只有 worker3 同时有):worker3 on-nvme nodeSelector disk=nvme(没有任何节点有):(未调度);0/5 nodes are available: 1 node(s) had untolerated taint(s), 4 node(s) didn't match Pod's node affinity/selector. preemption: 0/5 nodes are available: 5 Preemption is not 给 worker 打上 disk=nvme 之后:on-nvme 在 0.2 秒内被调度到 worker(调度器监听节点变化,不用重建 Pod) 把 worker3 的 disk 改成 hdd 之后:已经调度的 on-ssd 在 worker3(标签变了也不会被赶走)
标签 disk=ssd(worker2、worker3)、gpu=yes(worker3)。nodeSelector: {disk: ssd} 调度到 worker3;{disk: ssd, gpu: "yes"}(多个键 AND)也到 worker3;disk: nvme 没有节点有,Pod 保持 Pending。给 worker 打上 disk=nvme 之后 0.2 秒就被调度过去,不需要重建 Pod——调度器监听节点变化。反过来,把 worker3 的 disk 改成 hdd,已经调度的 Pod 不会被赶走。
nodeName 直接指定:worker;phase=Running;事件里有 Scheduled 吗:False(没有经过调度器) nodeName 指向已经 cordon 的节点:worker;phase=Running nodeSelector 指向已经 cordon 的节点:(未调度);phase=Pending;0/5 nodes are available: 1 node(s) had untolerated taint(s), 1 node(s) were unschedulable, 3 node(s) didn't match Pod's node affinity/selector. preemption: 0/5 nodes are nodeName 指向的节点放不下(请求 100 核):phase=Failed;reason=OutOfcpu;message=Pod was rejected: Node didn't have enough resource: cpu, requested: 100000, used: 100, capacity: 10000
直接写 nodeName 的 Pod 不经过调度器(事件里没有 Scheduled):指向已 cordon 的节点照样 Running,而 nodeSelector 指向它就 Pending(1 node(s) were unschedulable)。指向的节点放不下时没人替你挑别的节点,kubelet 直接拒绝:phase=Failed、reason=OutOfcpu、Pod was rejected: Node didn't have enough resource: cpu, requested: 100000, used: 100, capacity: 10000(单位是毫核)。想固定节点用 nodeSelector 加 kubernetes.io/hostname。
need-label nodeSelector: {no-such: label} → 0/5 nodes are available: 1 node(s) had untolerated taint(s), 1 node(s) were unschedulable, 3 node(s) didn't match Pod's node affinity/selector. preemp
need-cpu resources: {requests: {cpu: "100"}} → 0/5 nodes are available: 1 node(s) had untolerated taint(s), 1 node(s) were unschedulable, 3 Insufficient cpu. preemption: 0/5 nodes are available: 5
need-mem resources: {requests: {memory: 1000Gi}} → 0/5 nodes are available: 1 node(s) had untolerated taint(s), 1 node(s) were unschedulable, 3 Insufficient memory. preemption: 0/5 nodes are available:
need-eph resources: {requests: {ephemeral-storage: 1000Gi}} → 0/5 nodes are available: 1 node(s) had untolerated taint(s), 1 node(s) were unschedulable, 3 Insufficient ephemeral-storage. preemption: 0/5 nodes are
on-cordon nodeSelector: {kubernetes.io/hostname: kx-sched-worker4} → 0/5 nodes are available: 1 node(s) had untolerated taint(s), 1 node(s) were unschedulable, 3 node(s) didn't match Pod's node affinity/selector. preemp
need-cp nodeSelector: {node-role.kubernetes.io/control-plane: ""} → 0/5 nodes are available: 1 node(s) had untolerated taint(s), 1 node(s) were unschedulable, 3 node(s) didn't match Pod's node affinity/selector. preempmessage 是 0/5 nodes are available: 1 node(s) had untolerated taint(s), 1 node(s) were unschedulable, 3 …,最后一项因 Pod 而异:标签不匹配是 didn't match Pod's node affinity/selector,资源不足是 Insufficient cpu、Insufficient memory 或 Insufficient ephemeral-storage。每个节点只被计入第一个淘汰它的过滤器,所以各项加起来正好是 5(这里 worker4 被 cordon,控制面有污点,三个正常 worker 才轮到具体原因)。