京公网安备11010502063487号命名空间删了半天还是 Terminating,到底卡在哪一步?namespace 控制器把每一步的状态都写进了 status.conditions。这篇制造两种典型的卡住原因,读出对应的 condition,再试一次危险的“强制 finalize”,看它留下了什么后果。
kubectl delete ns k14a --wait=false:namespace "k14a" deleted
6 秒后 kubectl get ns k14a:NAME STATUS AGE
k14a Terminating 6s
status.phase=Terminating;spec.finalizers=['kubernetes'];deletionTimestamp=有
condition NamespaceContentRemaining: status=True reason=SomeResourcesRemain message=Some resources are remaining: configmaps. has 1 resource instances
condition NamespaceDeletionContentFailure: status=False reason=ContentDeleted message=All content successfully deleted, may be waiting on finalization
condition NamespaceDeletionDiscoveryFailure: status=False reason=ResourcesDiscovered message=All resources successfully discovered
condition NamespaceDeletionGroupVersionParsingFailure: status=False reason=ParsedGroupVersions message=All legacy kube types successfully parsed
condition NamespaceFinalizersRemaining: status=True reason=SomeFinalizersRemain message=Some content in the namespace has finalizers remaining: example.com/block in 1 resource instances删命名空间只是把 phase 设为 Terminating、spec.finalizers 里留着 kubernetes,然后由 namespace 控制器逐个清理里面的资源。ConfigMap blocker 带着自己的 finalizer,6 秒后命名空间还是 Terminating。kubectl get ns -o yaml 里 status.conditions 说得很清楚:NamespaceContentRemaining(Some resources are remaining: configmaps. has 1 resource instances)和 NamespaceFinalizersRemaining(example.com/block in 1 resource instances)为 True。
摘掉 blocker 的 finalizer 之后,命名空间 在 4.2 秒内消失
把 blocker 的 finalizer 摘掉,namespace 控制器清空内容后自己摘掉 kubernetes,命名空间在 4.2 秒内消失,全程没有动命名空间本身。排查原则:先看 condition,找到是谁在阻塞。
kubectl get apiservice v1beta1.k14.example.com:NAME SERVICE AVAILABLE AGE | v1beta1.k14.example.com default/no-such-svc False (ServiceNotFound) 4s
命名空间 k14b 里只有一个普通 ConfigMap,没有 finalizer;8 秒后 phase=Terminating
condition NamespaceContentRemaining: status=False reason=ContentRemoved message=All content successfully removed
condition NamespaceDeletionContentFailure: status=False reason=ContentDeleted message=All content successfully deleted, may be waiting on finalization
condition NamespaceDeletionDiscoveryFailure: status=True reason=DiscoveryFailed message=Discovery failed for some groups, 1 failing: unable to retrieve the complete list of server APIs: k14.example.com/v1beta1: stale GroupVersion discovery: k14.example.com/v1beta1
condition NamespaceDeletionGroupVersionParsingFailure: status=False reason=ParsedGroupVersions message=All legacy kube types successfully parsed
condition NamespaceFinalizersRemaining: status=False reason=ContentHasNoFinalizers message=All content-preserving finalizers finished
删掉这个 APIService 之后,k14b 在 0.6 秒内消失命名空间里只有一个普通 ConfigMap,没有任何 finalizer,也会卡住:集群里有一个 APIService 指向不存在的 Service(False (ServiceNotFound)),namespace 控制器枚举不全所有 API 组,不敢确认内容已清空,NamespaceDeletionDiscoveryFailure 变成 True。删掉这个 APIService,命名空间 0.6 秒内就消失了。生产里常见的是 metrics-server 之类聚合 API 挂掉。
kubectl replace --raw /api/v1/namespaces/k14c/finalize(把 spec.finalizers 清空):退出码 0;命名空间还在吗 False etcd 里 /registry/configmaps/k14c/ 下还剩:['/registry/configmaps/k14c/blocker', '/registry/configmaps/k14c/kube-root-ca.crt'] 重新创建同名命名空间 k14c 后 kubectl get cm -n k14c:NAME DATA AGE | blocker 0 6s | kube-root-ca.crt 1 6s
网上常见的“把命名空间的 spec.finalizers 清空再 PUT 到 /finalize”确实能让命名空间立刻消失,但它绕过了清理:etcd 里 /registry/configmaps/k14c/ 下的 blocker 和 kube-root-ca.crt 还在,重新创建同名命名空间,kubectl get cm 又看见它们。这是把症状按下去而不是解决原因。