京公网安备11010502063487号节点离开集群有两种:短暂离线(进程崩溃再重启)和永久离开(机器换掉、数据丢了)。它们在集群眼里差别很大:前者节点 ID 还在、自动回来;后者留下一个永远连不上的「幽灵」,需要 CLUSTER FORGET 清理。这篇把四种情形分别真做一遍,并纠正我一个预期:FORGET 在这个版本里居然会自己传播。
从库 7463 被 kill -9,4222 ms 后在其它节点眼中变成:slave,fail;集群状态(旁观节点): cluster_state:ok
重启后 63 ms:link up,在其它节点眼中 = slave;节点 ID 不变(仍是 3bcc524f);DBSIZE 从库/主库 = 8327 / 8327
主库同步统计变化:full +0,partial_ok +1
主库日志:
Partial resynchronization not accepted: Replication ID mismatch (Replica asked for '37ae532f0408b5156e751c83a4bac380df432cfc', my replication IDs are
Full resync requested by replica 127.0.0.1:7463 (rdb-channel)
Starting BGSAVE for SYNC with target: replicas sockets (rdb-channel)
Synchronization with replica 127.0.0.1:7463 succeeded
Partial resynchronization request from 127.0.0.1:7463 accepted. Sending 300704 bytes of backlog starting from offset 1.
从库日志:
Before turning into a replica, using my own master parameters to synthesize a cached master: I may be able to synchronize with the new master with jus
Connecting to MASTER 127.0.0.1:7460
MASTER <-> REPLICA sync started
Non blocking connect for SYNC fired the event.
Master replied to PING, replication can continue...
Trying a partial resynchronization (request 37ae532f0408b5156e751c83a4bac380df432cfc:1).
Node 5455a38d7b7a7dcb9fb75cab9c732728dea1129f () is no longer master of shard 3a96fa47a0476beb1885e939f076f70ee8b8a5c5; removed all 0 slot(s) it used
PSYNC is not possible, initialize RDB channel.
Node fdc50d7c067c35ad2525866129f374f856e85b49 () is no longer master of shard e0b2bceea5476937ae0c77d3a34446b29b5bcee8; removed all 0 slot(s) it used
MASTER <-> REPLICA sync: receiving streamed RDB from master with EOF to disk
Discarding previously cached master state.
MASTER <-> REPLICA sync: Loading DB in memory
MASTER <-> REPLICA sync: Flushing old data
MASTER <-> REPLICA sync: Finished with success
MASTER <-> REPLICA sync: Starting to stream replication buffer into the db (0 bytes).
MASTER <-> REPLICA sync: Successfully streamed replication buffer into the db (0 bytes in total)
Before turning into a replica, using my own master parameters to synthesize a cached master: I may be able to synchronize with the new master with jus
Discarding previously cached master state.
Before turning into a replica, using my own master parameters to synthesize a cached master: I may be able to synchronize with the new master with jus
Connecting to MASTER 127.0.0.1:7460
MASTER <-> REPLICA sync started
Non blocking connect for SYNC fired the event.
Master replied to PING, replication can continue...
Trying a partial resynchronization (request 2d72d92db74ae2bae7e5f64467fb283e3cef8e0e:1).
Successful partial resynchronization with master.
MASTER <-> REPLICA sync: Master accepted a Partial Resynchronization.kill -9 一个从库:约 4.2 s(cluster-node-timeout 3 s 之后)其它节点把它标成 slave,fail,但因为槽位都还有主库,cluster_state 仍是 ok。期间主库继续被写入 5000 个 key。重启后(nodes.conf 还在,节点 ID 不变)63 ms 就 link up 并追上了数据。日志的细节值得看:从库先尝试部分重同步,被主库以 Replication ID mismatch 拒绝,于是走 rdb-channel 的全量同步;之后又有一次 Partial resynchronization request … accepted. Sending 300704 bytes of backlog,用来补 RDB 之后的增量。所以主库 INFO stats 的变化是 sync_full +0、sync_partial_ok +1——和「发生过一次全量同步」的事实不一致。在这个版本上判断同步类型,要看日志,别只看这两个计数。(这次的 rdb-channel 全量同步与第 53 篇里我用计数判断类型的场景不同;那里我又对照日志复核过,结论没有变。)
7464 新 ID 6955e682(旧 ID 5455a38d);它自己已知节点 1 个
其它节点眼中旧 ID:slave;已知节点数(含幽灵)= 6
> CLUSTER FORGET 5455a38d7b7a7dcb9fb75cab9c732728dea1129f
"OK"
+ 2 ms 各节点已知节点数(只在 7460 上执行过 FORGET): {7460: 5, 7461: 6, 7462: 6, 7463: 6, 7465: 6}
+ 297 ms 各节点已知节点数(只在 7460 上执行过 FORGET): {7460: 5, 7461: 6, 7462: 6, 7463: 5, 7465: 6}
+ 356 ms 各节点已知节点数(只在 7460 上执行过 FORGET): {7460: 5, 7461: 5, 7462: 6, 7463: 5, 7465: 6}
+ 706 ms 各节点已知节点数(只在 7460 上执行过 FORGET): {7460: 5, 7461: 5, 7462: 5, 7463: 5, 7465: 6}
+ 761 ms 各节点已知节点数(只在 7460 上执行过 FORGET): {7460: 5, 7461: 5, 7462: 5, 7463: 5, 7465: 5}
旧 ID 在 7462 眼中 = (不在节点表里)把一个从库的 nodes.conf 和数据文件删掉再启动,它就是一个全新的节点:新的 ID,只认识自己。而其它节点还记着旧 ID(已知节点数仍是 6)。我另外用一个不做任何操作的对照观察了 22 秒(该次探针没有打印进上面的输出):这个幽灵一直是 slave/disconnected,不会自己消失。在一个节点上 CLUSTER FORGET 旧ID 后,该节点立刻少一个;其它节点在 761 ms 内也都删除了它——我原本以为要在每个节点上都执行,实测这个版本里 FORGET 会通过集群总线传播,我没有去查是从哪个版本开始的。
> CLUSTER MEET 127.0.0.1 7460 "OK" MEET 后 235 ms 认识全部 6 个节点 > CLUSTER REPLICATE a16642ad99c3e50b028d187cc286b71e495c3158 "OK" REPLICATE 后 40 ms:link up,DBSIZE = 8334,角色 = slave
CLUSTER MEET 一个已有节点,1291 ms 后认识全部 6 个节点(第 74 篇);再 CLUSTER REPLICATE 主库ID,41 ms 后从库 link up、数据同步完成。
7462 被杀,7465 在 5187 ms 后升主 旧主库重启后 100 ms:角色 = slave,master = 7465(新主库端口 7465),ID 不变 f2507670
主库被杀,从库 5187 ms 后升主;旧主库用原来的 nodes.conf 重启后 100 ms 内自动变成新主库的从库,节点 ID 不变。不需要 CLUSTER REPLICATE——节点重启后从 gossip 里看到更高的 epoch,自己就明白了(第 77 篇)。