京公网安备11010502063487号Pod 卡在 ImagePullBackOff 到底是什么意思?怎么区分是网络问题、认证问题还是镜像根本不存在?kubelet 多久重试一次?这篇用一个假仓库制造四种拉取失败和一种配置错误,再把 330 秒里的重试节奏逐个记下来。
refused localhost:5999/nolocal:1.0 → ErrImagePull;原因:仓库地址没有服务(连接被拒绝)
message:failed to resolve reference "localhost:5999/nolocal:1.0": failed to do request: Head "https://localhost:5999/v2/nolocal/manifests/1.0": dial tcp [::1]:5999: connect: connection refused
notfound localhost:5000/missing:1.0 → ErrImagePull;原因:仓库在线,镜像或 tag 不存在(404)
message:rpc error: code = NotFound desc = failed to resolve reference "localhost:5000/missing:1.0": localhost:5000/missing:1.0: not found
private localhost:5000/private/app:1.0 → ErrImagePull;原因:需要认证(401)
message:failed to resolve reference "localhost:5000/private/app:1.0": unexpected status from HEAD request to http://localhost:5000/v2/private/app/manifests/1.0: 401 Unauthorized
nohost nonexistent.invalid/app:1.0 → ErrImagePull;原因:域名解析失败
message:failed to resolve reference "nonexistent.invalid/app:1.0": failed to do request: Head "https://nonexistent.invalid/v2/app/manifests/1.0": dial tcp: lookup nonexistent.invalid on <本机局域网IP>:53: no such host
badname busybox:1.36:extra → InvalidImageName;原因:镜像名格式不对
message:Failed to apply default image tag "busybox:1.36:extra": couldn't parse image name "busybox:1.36:extra": invalid reference format在 worker 上用 hostNetwork Pod 跑了一个假的镜像仓库(localhost:5000:/v2/ 返回 200,名字里有 private 的返回 401,其余 manifest 返回 404),让五个 Pod 各撞上一种失败:localhost:5999 没有服务 → Head "https://localhost:5999/v2/nolocal/manifests/1.0": dial tcp [::1]:5999: connect: connection refused;localhost:5000/missing:1.0 → code = NotFound … not found;localhost:5000/private/app:1.0 → unexpected status from HEAD request to http://localhost:5000/v2/private/app/manifests/1.0: 401 Unauthorized;nonexistent.invalid/app:1.0 → lookup nonexistent.invalid … no such host;busybox:1.36:extra → InvalidImageName(couldn't parse image name … invalid reference format)。(在另一次运行里还看到:同一个 Pod 进入 ImagePullBackOff 后 message 变成 Back-off pulling image …: ErrImagePull: <真正的原因>,原因仍在后面。)
Pod 创建后 330 秒内,容器状态变化(相对秒, reason):[(0, 'ContainerCreating'), (1, 'ErrImagePull'), (13, 'ImagePullBackOff'), (24, 'ErrImagePull'), (36, 'ImagePullBackOff'), (48, 'ErrImagePull'), (63, 'ImagePullBackOff'), (103, 'ErrImagePull'), (116, 'ImagePullBackOff'), (194, 'ErrImagePull'), (208, 'ImagePullBackOff')]
每次出现 ErrImagePull(= 一次新的拉取尝试失败)的时刻:[1, 24, 48, 103, 194];相邻尝试的间隔(秒):[23, 24, 55, 91]
假仓库日志里 missing/manifests/2.0 的请求共 6 条
事件(去重后):[('BackOff', 21), ('Failed', 2), ('Failed', 3), ('Failed', 5), ('Failed', 21), ('Pulling', 5)]对 localhost:5000/missing:2.0 观察 330 秒:容器状态依次是 ContainerCreating(0 秒)→ ErrImagePull(1)→ ImagePullBackOff(13)→ ErrImagePull(24)→ ImagePullBackOff(36)→ ErrImagePull(48)→ ImagePullBackOff(63)→ ErrImagePull(103)→ ImagePullBackOff(116)→ ErrImagePull(194)→ ImagePullBackOff(208)。每次出现 ErrImagePull 就是一次新的拉取尝试(事件里 Pulling 共 5 次),相邻尝试的间隔是 23、24、55、91 秒——越来越长,大致是 10、20、40、80 秒的退避再加上 kubelet 同步周期带来的十几秒抖动。修好问题之后,要等当前这一轮退避结束才会重试。