Redis Sentinel 在重启之前不会复活 master -> slave

Question

我在使用 Sentinel 恢复主节点时遇到问题。具体来说，当 master 丢失时，slaves 会被正确提升，但 master 在重新启动时永远不会降级。但是，如果我立即重新启动 Sentinel，主节点将被降级。是我的配置不好，还是我缺少一些基本的东西？

编辑:Xpost https://groups.google.com/forum/#!topic/redis-db/4AnGNssqYTw

我如下设置了几个虚拟机，全部使用 Redis 3.1.999:

192.168.0.101 - Redis Slave
192.168.0.102 - Redis Slave
192.168.0.103 - Redis Master
192.168.0.201 - Sentinel
192.168.0.202 - Sentinel

我的哨兵配置，两个哨兵:

loglevel verbose
logfile "/tmp/sentinel.log"
sentinel monitor redisA01 192.168.0.101 6379 2
sentinel down-after-milliseconds redisA01 30000
sentinel failover-timeout redisA01 120000

我在主节点上停止了redis；正如预期的那样，Sentinel 捕获它并将一个 slave 提升为 master。

3425:X 08 Sep 23:47:43.839 # +sdown master redisA01 192.168.0.103 6379
3425:X 08 Sep 23:47:43.896 # +odown master redisA01 192.168.0.103 6379 #quorum 2/2
3425:X 08 Sep 23:47:43.896 # +new-epoch 53
3425:X 08 Sep 23:47:43.896 # +try-failover master redisA01 192.168.0.103 6379
3425:X 08 Sep 23:47:43.898 # +vote-for-leader 71de0d8f6250e436e1f76800cbe8cbae56c1be7c 53
3425:X 08 Sep 23:47:43.901 # 192.168.0.201:26379 voted for 71de0d8f6250e436e1f76800cbe8cbae56c1be7c 53
3425:X 08 Sep 23:47:43.975 # +elected-leader master redisA01 192.168.0.103 6379
3425:X 08 Sep 23:47:43.976 # +failover-state-select-slave master redisA01 192.168.0.103 6379
3425:X 08 Sep 23:47:44.077 # +selected-slave slave 192.168.0.102:6379 192.168.0.102 6379 @ redisA01 192.168.0.103 6379
3425:X 08 Sep 23:47:44.078 * +failover-state-send-slaveof-noone slave 192.168.0.102:6379 192.168.0.102 6379 @ redisA01 192.168.0.103 6379
3425:X 08 Sep 23:47:44.977 * +failover-state-wait-promotion slave 192.168.0.102:6379 192.168.0.102 6379 @ redisA01 192.168.0.103 6379
3425:X 08 Sep 23:47:44.980 - -role-change slave 192.168.0.102:6379 192.168.0.102 6379 @ redisA01 192.168.0.103 6379 new reported role is master
3425:X 08 Sep 23:47:44.981 # +promoted-slave slave 192.168.0.102:6379 192.168.0.102 6379 @ redisA01 192.168.0.103 6379
3425:X 08 Sep 23:47:44.981 # +failover-state-reconf-slaves master redisA01 192.168.0.103 6379
3425:X 08 Sep 23:47:45.068 * +slave-reconf-sent slave 192.168.0.101:6379 192.168.0.101 6379 @ redisA01 192.168.0.103 6379
3425:X 08 Sep 23:47:46.031 * +slave-reconf-inprog slave 192.168.0.101:6379 192.168.0.101 6379 @ redisA01 192.168.0.103 6379
3425:X 08 Sep 23:47:46.032 * +slave-reconf-done slave 192.168.0.101:6379 192.168.0.101 6379 @ redisA01 192.168.0.103 6379
3425:X 08 Sep 23:47:46.101 # -odown master redisA01 192.168.0.103 6379
3425:X 08 Sep 23:47:46.101 # +failover-end master redisA01 192.168.0.103 6379
3425:X 08 Sep 23:47:46.102 # +switch-master redisA01 192.168.0.103 6379 192.168.0.102 6379
3425:X 08 Sep 23:47:46.103 * +slave slave 192.168.0.101:6379 192.168.0.101 6379 @ redisA01 192.168.0.102 6379
3425:X 08 Sep 23:47:46.103 * +slave slave 192.168.0.103:6379 192.168.0.103 6379 @ redisA01 192.168.0.102 6379

我等了几分钟，然后在之前的主节点上重启了Redis。出乎意料的是(对我来说)节点没有降级为从节点。

3425:X 08 Sep 23:48:16.105 # +sdown slave 192.168.0.103:6379 192.168.0.103 6379 @ redisA01 192.168.0.102 6379
3425:X 08 Sep 23:50:09.131 # -sdown slave 192.168.0.103:6379 192.168.0.103 6379 @ redisA01 192.168.0.102 6379

再等几分钟后，我重新启动了其中一个哨兵。它立即检测到悬挂的前主节点并将其降级。

3425:signal-handler (1441758237) Received SIGTERM scheduling shutdown...
...
3670:X 09 Sep 00:23:57.687 # Sentinel ID is 71de0d8f6250e436e1f76800cbe8cbae56c1be7c
3670:X 09 Sep 00:23:57.687 # +monitor master redisA01 192.168.0.102 6379 quorum 2
3670:X 09 Sep 00:23:57.690 - -role-change slave 192.168.0.103:6379 192.168.0.103 6379 @ redisA01 192.168.0.102 6379 new reported role is master
3670:X 09 Sep 00:23:58.708 - Accepted 192.168.0.201:49731
3670:X 09 Sep 00:24:07.778 * +convert-to-slave slave 192.168.0.103:6379 192.168.0.103 6379 @ redisA01 192.168.0.102 6379
3670:X 09 Sep 00:24:17.801 - +role-change slave 192.168.0.103:6379 192.168.0.103 6379 @ redisA01 192.168.0.102 6379 new reported role is slave

Answer 1

我会检查主机上的多个进程，以及可能的循环复制。如果您查看第一个日志批处理的末尾，您会看到它已经通过 +slave 条目将 103 IP 检测为从站。我会试着看看为什么新主人在晋升时已经把旧主人当作奴隶。

根据提供的日志，在重新启动时重新配置正在发生，在从属重新发现之前它检测到从属将自己报告为主控。

再试一次，它在重新启动sentinel之前直接询问每个节点，看看他们各自有什么master和slaves。这可能会阐明根本问题。

编辑:您描述的哨兵配置不正确。您在 list 中将主人列为 103，但您发布的哨兵配置文件指示 101，根据您的 list ，这是一个奴隶。

此外，添加第三个哨兵。两个使脑裂变得容易，这很可能是您所看到的。