京公网安备11010502063487号故障转移的关键是「多数主库同意」。但这个「多数」在哪一步起作用、谁有投票权、什么时候即使没有选举也不可能成功?这篇用 CLUSTER INFO 里的投票消息计数、节点日志和角色表,把三种情形的每一票都数出来。
被杀主库 [7400];升主的从库及耗时(ms):{7405: 5860};cluster_state = ok
各幸存节点 CLUSTER INFO 里投票消息数的变化(auth-req = 要票,auth-ack = 投票):
7401 master {'auth-req_sent': 0, 'auth-req_received': 1, 'auth-ack_sent': 1, 'auth-ack_received': 0}
7402 master {'auth-req_sent': 0, 'auth-req_received': 1, 'auth-ack_sent': 1, 'auth-ack_received': 0}
7403 slave {'auth-req_sent': 0, 'auth-req_received': 1, 'auth-ack_sent': 0, 'auth-ack_received': 0}
7404 slave {'auth-req_sent': 0, 'auth-req_received': 1, 'auth-ack_sent': 0, 'auth-ack_received': 0}
7405 master {'auth-req_sent': 5, 'auth-req_received': 0, 'auth-ack_sent': 0, 'auth-ack_received': 2}
从库的日志:
FAIL message received from 8c092c77ba51d49d111c51aa437c7ef25095ed44 () about dcb5ace1dc91802a07af4d4f49b5abcd77eea6fc ()
Start of election delayed for 878 milliseconds (rank #0, offset 0).
Starting a failover election for epoch 7.
Failover election won: I'm the new master.
投票的主库的日志:
[7401] Failover auth granted to 9c42c80237d022b44c51b2df7fcecc0f48efcfd2 () for epoch 7
[7402] Failover auth granted to 9c42c80237d022b44c51b2df7fcecc0f48efcfd2 () for epoch 7从库看到 FAIL 之后先延迟 878 ms(延迟 = 固定 500 ms + 随机 + 排名相关,这里是 rank #0——这个公式来自文档,我只看到了日志里的这个数),然后发起选举,Starting a failover election for epoch 7。它向所有节点发出 auth 请求(5 条),收到 2 票:两个幸存主库各给一票(日志 Failover auth granted to … for epoch 7)。两个从库虽然也收到了请求,但没有投票。3 个主库的多数是 2,拿到 2 票就赢,Failover election won。
被杀主库 [7410, 7411];15 秒内升主的从库:无;cluster_state = fail
各幸存节点的角色: {7415: 'slave', 7414: 'slave', 7413: 'slave', 7410: 'master,fail?', 7411: 'master,fail?', 7412: 'master'}
幸存节点上投票消息数的变化:
7412 master {'auth-req_sent': 0, 'auth-req_received': 0, 'auth-ack_sent': 0, 'auth-ack_received': 0}
7413 slave {'auth-req_sent': 0, 'auth-req_received': 0, 'auth-ack_sent': 0, 'auth-ack_received': 0}
7414 slave {'auth-req_sent': 0, 'auth-req_received': 0, 'auth-ack_sent': 0, 'auth-ack_received': 0}
7415 slave {'auth-req_sent': 0, 'auth-req_received': 0, 'auth-ack_sent': 0, 'auth-ack_received': 0}
从库的日志(节选):这是最容易被误解的一种:3 主 3 从里同时杀掉 2 个主库,你可能以为「从库发起选举、但拿不到票」。实际上一条 auth 请求都没有发出——所有幸存节点的 auth-req 计数全是 0。看幸存节点眼中被杀主库的角色:master,fail?,一直是 PFAIL,没有变成 fail。我推断原因是:把 PFAIL 确认成 FAIL 需要多数主库同意,3 个主库只剩 1 个凑不齐,从库只在看到 FAIL 后才发起选举。这个「需要多数主库」的机制是官方设计说明,本次实验的证据是「没有 FAIL、没有选举」这个现象,我没有再去区分其它可能的原因。结果:fail,15 秒里没有任何从库升主。
被杀主库 [7420, 7421];升主的从库及耗时(ms):{7425: 5269, 7427: 5619};cluster_state = ok
幸存节点上投票消息数的变化:
7422 master {'auth-req_sent': 0, 'auth-req_received': 2, 'auth-ack_sent': 2, 'auth-ack_received': 0}
7423 master {'auth-req_sent': 0, 'auth-req_received': 2, 'auth-ack_sent': 2, 'auth-ack_received': 0}
7424 master {'auth-req_sent': 0, 'auth-req_received': 2, 'auth-ack_sent': 2, 'auth-ack_received': 0}
7425 master {'auth-req_sent': 9, 'auth-req_received': 1, 'auth-ack_sent': 1, 'auth-ack_received': 3}
7426 slave {'auth-req_sent': 0, 'auth-req_received': 2, 'auth-ack_sent': 0, 'auth-ack_received': 0}
7427 master {'auth-req_sent': 9, 'auth-req_received': 1, 'auth-ack_sent': 0, 'auth-ack_received': 4}
7428 slave {'auth-req_sent': 0, 'auth-req_received': 2, 'auth-ack_sent': 0, 'auth-ack_received': 0}
7429 slave {'auth-req_sent': 0, 'auth-req_received': 2, 'auth-ack_sent': 0, 'auth-ack_received': 0}杀 2 个主库后还剩 3 个主库——5 个的多数。两个失去主库的从库都成功升主(kill 后 5269 ms 和 5619 ms),cluster_state = ok。三个幸存主库每个都投出了 2 票(auth-ack_sent = 2,分别投给两个候选);先升主的从库随后也对另一个候选投了票(它的 auth-ack_sent = 1),推测是因为它此时已经是主库,有了投票权。
三个观察合起来:投票权只属于主库(从库不投票);多数是相对所有主库的多数(3 主要 2、5 主要 3);而一旦幸存主库不足多数,连「确认主库已死」这一步都过不去,选举都不会开始。所以规划集群时,主库数量决定了能同时容忍几个主库宕机(3 主 → 1 个,5 主 → 2 个),从库数量只决定每个分片有没有可升的候选。