京公网安备11010502063487号多副本的控制器怎么保证同一时刻只有一个在干活?答案是一个叫 Lease 的 API 对象。这篇不看文档,直接让三个真实的 kube-scheduler 争一把锁,然后用 kill -9、force delete、普通删除和篡改 Lease 四种方式弄掉 leader,量出每种情况下换人要多久、Lease 里发生了什么。
kube-controller-manager:holderIdentity=kx-main-control-plane_20b936db-e39b-4079-989e-e33644686e8a leaseDurationSeconds=15 leaseTransitions=0;9 秒内观察到 6 次续约,相邻两次 renewTime 间隔 [2.0, 2.0, 2.0, 2.0, 2.0] 秒 kube-scheduler:holderIdentity=kx-main-control-plane_78a75147-e8dd-4964-9cbb-3ffe8be985c2 leaseDurationSeconds=15 leaseTransitions=0;9 秒内观察到 5 次续约,相邻两次 renewTime 间隔 [2.0, 2.0, 2.0, 2.0] 秒
选主用的“锁”是 coordination.k8s.io 里的 Lease 对象:holderIdentity(主机名加随机 uuid)是当前持有者,renewTime 是它最后一次续约,leaseDurationSeconds 是别人要等多久才能抢,leaseTransitions 记录换过几次主。实测 kube-controller-manager 和 kube-scheduler 的 Lease 每 2.0 秒续约一次,leaseDurationSeconds=15。
Lease:holderIdentity=sched-a_0f50a938-c557-48d3-9b89-fa313786d39a;leaseDurationSeconds=15;leaseTransitions=0;acquireTime=2026-10-10T09:03:03.226351Z
sched-a 的日志:I1010 09:03:03.226336 1 leaderelection.go:258] "Attempting to acquire leader lease..." lock="k15/k15-sched"
sched-a 的日志:I1010 09:03:03.230680 1 leaderelection.go:272] "Successfully acquired lease" lock="k15/k15-sched"
sched-b 的日志:I1010 09:03:04.438823 1 leaderelection.go:258] "Attempting to acquire leader lease..." lock="k15/k15-sched"
sched-c 的日志:I1010 09:03:04.485850 1 leaderelection.go:258] "Attempting to acquire leader lease..." lock="k15/k15-sched"用同样的 kube-scheduler 镜像再起三个副本(自定义调度器名 k15-scheduler 与 Lease k15/k15-sched,不影响默认调度器)。先启动的 sched-a 日志里 Attempting to acquire leader lease… 后紧跟 Successfully acquired lease;sched-b、sched-c 只有前一句,一直在重试。选主就是抢一个对象的 holderIdentity,抢到的人负责续约。
① 在节点上对 leader 进程 kill -9(没有机会释放租约):原 leader=sched-a(被动手时离上次续约约 1.5 秒)
Lease 状态变化:[+0.0s holder=sched-a leaseDuration=15] → [+16.6s holder=sched-b leaseDuration=15]
新 leader=sched-b,接手用了 16.6 秒;leaseTransitions 0 → 1
② kubectl delete pod --grace-period=0 --force(API 对象立刻删掉,kubelet 仍会终止进程):原 leader=sched-b(被动手时离上次续约约 1.2 秒)
Lease 状态变化:[+0.0s holder=(空) leaseDuration=1] → [+2.5s holder=sched-a leaseDuration=15]
新 leader=sched-a,接手用了 2.5 秒;leaseTransitions 1 → 2
③ kubectl delete pod(默认宽限期,进程收到 SIGTERM):原 leader=sched-a(被动手时离上次续约约 0.9 秒)
Lease 状态变化:[+0.1s holder=(空) leaseDuration=1] → [+1.1s holder=sched-c leaseDuration=15]
新 leader=sched-c,接手用了 1.1 秒;leaseTransitions 2 → 3三种让 leader 消失的方式差别很大。在节点上对进程 kill -9,它没机会释放租约,sched-b 要等租约过期:16.6 秒。kubectl delete pod --grace-period=0 --force 只是让 API 对象立刻消失,kubelet 仍会终止进程,进程退出前把 Lease 置成 holderIdentity 为空、leaseDurationSeconds=1(释放),别的候选者 2.5 秒就接手;普通删除(SIGTERM)同样释放,1.1 秒接手。所以“滚动升级时 leader 切换很快、节点突然掉电时要十几秒”是正常现象。
把 Lease 的 holderIdentity 改成 intruder(当前 leader 是 sched-c):lease.coordination.k8s.io/k15-sched patched
sched-c 的容器重启次数 0 → 1(12.9 秒后重启:True)
sched-c 重启前的日志:E1010 09:03:55.382179 1 leaderelection.go:473] "Error retrieving lease lock" err="context deadline exceeded" lock="k15/k15-sched"
sched-c 重启前的日志:I1010 09:03:55.382351 1 leaderelection.go:304] "Failed to renew lease" lock="k15/k15-sched" err="context deadline exceeded"
sched-c 重启前的日志:I1010 09:03:55.387038 1 requestheader_controller.go:194] Shutting down RequestHeaderAuthRequestController
sched-c 重启前的日志:E1010 09:03:55.387304 1 server.go:337] "Leaderelection lost"
intruder 一直占着 Lease,直到 6.0 秒后 holderIdentity 才变成 sched-b(从我改写 Lease 起算 19.1 秒);leaseTransitions=4把 holderIdentity 改成 intruder 后,当前 leader sched-c 的续约失败(Failed to renew lease … context deadline exceeded),随即 Leaderelection lost、进程退出,容器 12.9 秒后重启——leader 一旦发现自己不再持有锁就停止工作,而不是继续。新 leader 是 sched-b,在改写之后 19.1 秒才接手:它必须自己观察满一个 leaseDuration(15 秒)没有变化才敢抢,用的是自己本地的时钟,而不是 renewTime 里的时间戳。