京公网安备11010502063487号集群满了,高优先级的 Pod 怎么挤进去?谁会被牺牲?要等多久?这篇用 requests.cpu 把两个节点占满,让高优先级 Pod 触发抢占,把提名、受害者的终止、事件和 Pod 条件的时间线量出来,再看 preemptionPolicy: Never 的行为。
NAME VALUE GLOBAL-DEFAULT PREEMPTION k36-high 1000 <none> PreemptLowerPriority k36-high-never 1000 <none> Never k36-low 100 <none> PreemptLowerPriority k36-mid 500 <none> PreemptLowerPriority system-cluster-critical 2000000000 <none> PreemptLowerPriority system-node-critical 2000001000 <none> PreemptLowerPriority Pod p-none:priorityClassName=None → spec.priority=0;preemptionPolicy=PreemptLowerPriority Pod p-high:priorityClassName=k36-high → spec.priority=1000;preemptionPolicy=PreemptLowerPriority priorityClassName 写了不存在的类:退出码 1;Error from server (Forbidden): error when creating "STDIN": pods "p-bad" is forbidden: no PriorityClass with name no-such-class was found
PriorityClass 是名字到整数的映射,准入时写进 pod.spec.priority:没写 priorityClassName 的 Pod 是 0,k36-high 是 1000,系统自带的 system-cluster-critical 和 system-node-critical 是 20 亿级。写了不存在的类,创建被拒绝(no PriorityClass with name … was found)。每个类还带 preemptionPolicy,默认 PreemptLowerPriority。
4 个 low Pod(requests.cpu=4,优先级 100,terminationGracePeriodSeconds=12)分布:low1→worker2, low2→worker, low3→worker2, low4→worker
+ 0.0s hi phase=Pending nominatedNodeName=worker2
+ 0.0s low3 phase=Running 正在被删除(deletionTimestamp 已设置)
+ 12.4s hi phase=Pending
+ 12.4s low3 phase=Failed 正在被删除(deletionTimestamp 已设置)
+ 13.1s hi phase=Running
事件 low1 Scheduled Successfully assigned k36/low1 to kx-main-worker2
事件 low2 Scheduled Successfully assigned k36/low2 to kx-main-worker
事件 low3 Scheduled Successfully assigned k36/low3 to kx-main-worker2
事件 low4 Scheduled Successfully assigned k36/low4 to kx-main-worker
事件 hi FailedScheduling 0/3 nodes are available: 1 node(s) had untolerated taint(s), 2 Insufficient cpu. preemption: found a potential placement for pod on node kx-
事件 low3 Preempted Preempted by pod 2b3c8351-8a7e-40b6-985b-4f78db751ab6 on node kx-main-worker2
事件 low3 Killing Stopping container c
事件 hi Scheduled Successfully assigned k36/hi to kx-main-worker2
最终:hi 在 worker2 上 phase=Running;被抢占的 low Pod(取自 Preempted 事件):['low3'];hi 用了 13.1 秒才 Running(受害者的宽限期是 12 秒)
受害者 low3 的状态:phase=Failed;conditions=[('DisruptionTarget', 'True', 'PreemptionByScheduler')];message=['default-scheduler: preempting to accommodate a higher priority pod']两个 worker 各 10 核,4 个 requests.cpu=4 的低优先级 Pod 每个节点放了 2 个。高优先级 hi 要 4 核,没有节点放得下:FailedScheduling: 0/3 nodes are available: 1 node(s) had untolerated taint(s), 2 Insufficient cpu. preemption: found a potential placement …。随后 hi 的 status.nominatedNodeName 变成 worker2,同一时刻 worker2 上的 low3 被设上 deletionTimestamp,事件 Preempted;low3 有 12 秒宽限期,所以资源要到 +12.4 秒才释放,hi 在 +13.1 秒才真正 Running。受害者留下 DisruptionTarget 条件:PreemptionByScheduler: default-scheduler: preempting to accommodate a higher priority pod。
worker 上:a-low1、a-low2(优先级 100);worker2 上:b-mid1、b-mid2(优先级 500);hi(优先级 1000,requests.cpu=4)要么抢占 worker 上的一个 100,要么抢占 worker2 上的一个 500 结果:hi 在 worker;被抢占的(从 Pod 列表里消失或状态为 Failed):['a-low2'];仍在 Running 的:['a-low1', 'b-mid1', 'b-mid2']
worker 上是两个优先级 100 的 Pod,worker2 上是两个优先级 500 的 Pod,hi 抢占哪一边都能放下。结果 hi 到了 worker,被抢占的是 a-low2,优先级 500 的两个 Pod 毫发无损:调度器优先选择让优先级最低的 Pod 当受害者,并且只抢占优先级更低的。
hi-never(priority=1000,preemptionPolicy=Never):phase=Pending;PodScheduled=False reason=Unschedulable;nominatedNodeName=None
message=0/3 nodes are available: 1 node(s) had untolerated taint(s), 2 Insufficient cpu. preemption: not eligible due to preemptionPolicy=Never.
4 个 low Pod 都还在:['low1', 'low2', 'low3', 'low4']
手动删掉 low1 之后:hi-never phase=Running(优先级让它先于别的 Pending Pod 被调度,但没有抢占)preemptionPolicy: Never 的 hi-never(优先级同样是 1000)在集群被占满时保持 Pending,nominatedNodeName 为空,message 末尾是 preemption: not eligible due to preemptionPolicy=Never.,4 个低优先级 Pod 都还在;手动腾出资源之后它才被调度。它的优先级只影响排队顺序,不会去抢别人。适合“重要但可以等”的批处理任务。