Kafka 4.0 之前,ZooKeeper 是 Kafka 集群的「元数据库」。这篇真起一个 3 broker 的 Kafka 3.9.2(ZooKeeper 模式),看它在 ZooKeeper 里到底存了什么、没有存什么(比如消费 offset),以及 controller 被杀之后多久恢复。
顶层 znode: ['admin', 'brokers', 'cluster', 'config', 'consumers', 'controller', 'controller_epoch', 'feature', 'isr_change_notification', 'latest_producer_id_block', 'log_dir_event_notification']
/brokers/ids/1 = {"features":{},"listener_security_protocol_map":{"PLAINTEXT":"PLAINTEXT"},"endpoints":["PLAINTEXT://127.0.0.1:7901"],"jmx_port":-1,"port":7901,"host":"127.0.0.1","version":5,"timestamp":"1790833601663"} | 临时节点: True
/controller = {"version":2,"brokerid":1,"timestamp":"1790833601696","kraftControllerEpoch":-1} | 临时节点: True
/controller_epoch = 1 | /cluster/id = k9azSlstRb6suthYMk0Vdg
/brokers/ids 子节点: ['1', '2', '3']整个 controller 选举就是之前讲的 Leader 选举:抢 /controller 这个临时节点。
Created topic t1.
/brokers/topics/t1 = {"partitions":{"0":[3,1,2],"1":[1,2,3],"2":[2,3,1]},"topic_id":"R7nKRxIGQuGck7zs4kSy1A","adding_replicas":{},"removing_replicas":{},"version":3}
/brokers/topics/t1/partitions/0/state = {"controller_epoch":1,"leader":3,"version":1,"leader_epoch":0,"isr":[3,1,2]}
/brokers/topics/t1/partitions/1/state = {"controller_epoch":1,"leader":1,"version":1,"leader_epoch":0,"isr":[1,2,3]}
/brokers/topics/t1/partitions/2/state = {"controller_epoch":1,"leader":2,"version":1,"leader_epoch":0,"isr":[2,3,1]}
生产 1000 条、消费组 g1 消费 1000 条并提交 offset
ZooKeeper 里共 137 个 znode(不含 /zookeeper);其中路径含消费组名 g1 的:[];/consumers 的子节点:[](是空的;这个节点为什么还在,我没有查)
/brokers/topics 下的主题: ['__consumer_offsets', 't1'] (__consumer_offsets 是 Kafka 自己存 offset 的内部主题,不在 ZooKeeper 里存 offset) 被杀的是 controller = broker 1;之前各分区 leader {0: 3, 1: 1, 2: 2}
/controller 在 5.7 s 后变成 broker 2;/controller_epoch 1 → 2;t1 所有分区 leader 都离开了 broker 1:5.8 s 后;现在 {0: 3, 1: 2, 2: 2}
/brokers/ids 现在: ['2', '3']空窗由 broker 的会话超时决定,与会话过期那篇的结论一致。
重新启动 broker 1 后 /brokers/ids = ['1', '2', '3'] | t1 的 ISR = {0: [3, 2, 1], 1: [2, 3, 1], 2: [2, 3, 1]}zk_znode_count=142 zk_watch_count=21 zk_ephemerals_count=4 zk_num_alive_connections=5 zk_outstanding_requests=0
负载很小;但分区越多 znode 越多,watch 也越多,这与大量 znode那篇的结论方向相同。