redis集群部署——排错

redis集群部署时候，已经部署了集群（三主三从）
问题一

192.168.1.134:7000>  cluster nodes 
ba4f83ec20945617d089adf7db640a6052cb2fa5 192.168.1.133:7002@17002 slave 17ff68e0800d8aea279bddacf27c5d2d0cfb52a0 0 1654065263135 7 connected
e1804de236f04b121ffff5bb5c5e661adfa50c66 192.168.1.135:7004@17004 master - 0 1654065261124 5 connected 10923-16383
17ff68e0800d8aea279bddacf27c5d2d0cfb52a0 192.168.1.135:7005@17005 master - 0 1654065262000 7 connected 5461-10922
2f6d776193a48b4dd39c908a6f0cd7c99450746d 192.168.1.134:7001@17001 slave e1804de236f04b121ffff5bb5c5e661adfa50c66 0 1654065259000 5 connected
70d81f0d43f2bc131ad7356711c5b49d49d10507 192.168.1.133:7003@17003 slave 1731b6bbca92f945dd01cf589c8f0fbc5048205b 0 1654065262129 1 connected
1731b6bbca92f945dd01cf589c8f0fbc5048205b 192.168.1.134:7000@17000 myself,master - 0 1654065261000 1 connected 0-5460

但是因为一些误操作出现了以下问题：

192.168.1.134:7001> cluster info 
cluster_state:ok
cluster_slots_assigned:16384
cluster_slots_ok:16384
cluster_slots_pfail:0
cluster_slots_fail:0
cluster_known_nodes:5
cluster_size:3
cluster_current_epoch:8
cluster_my_epoch:5
cluster_stats_messages_ping_sent:60
cluster_stats_messages_pong_sent:66
cluster_stats_messages_sent:126
cluster_stats_messages_ping_received:66
cluster_stats_messages_pong_received:59
cluster_stats_messages_received:125
192.168.1.134:7001> cluster nodes
2f6d776193a48b4dd39c908a6f0cd7c99450746d 192.168.1.134:7001@17001 myself,slave e1804de236f04b121ffff5bb5c5e661adfa50c66 0 1654075034000 5 connected
ba4f83ec20945617d089adf7db640a6052cb2fa5 192.168.1.133:7002@17002 master - 0 1654075035373 8 connected 5461-10922
1731b6bbca92f945dd01cf589c8f0fbc5048205b 192.168.1.134:7000@17000 master - 0 1654075036000 1 connected 0-5460
70d81f0d43f2bc131ad7356711c5b49d49d10507 192.168.1.133:7003@17003 slave 1731b6bbca92f945dd01cf589c8f0fbc5048205b 0 1654075037000 1 connected
e1804de236f04b121ffff5bb5c5e661adfa50c66 192.168.1.135:7004@17004 master - 0 1654075037386 5 connected 10923-16383

三主三从变三主两从。192.168.1.133:7005不见了。
看7005的日志

[root@localhost redis]# tail -29 /var/log/redis_7005.log 
30878:M 01 Jun 2022 05:01:01.845 * Node configuration loaded, I'm 17ff68e0800d8aea279bddacf27c5d2d0cfb52a0
30878:M 01 Jun 2022 05:01:01.846 * Running mode=cluster, port=7005.
30878:M 01 Jun 2022 05:01:01.846 # WARNING: The TCP backlog setting of 511 cannot be enforced because /proc/sys/net/core/somaxconn is set to the lower value of 128.
30878:M 01 Jun 2022 05:01:01.846 # Server initialized
30878:M 01 Jun 2022 05:01:01.846 * Reading RDB preamble from AOF file...
30878:M 01 Jun 2022 05:01:01.846 * Loading RDB produced by version 6.2.0
30878:M 01 Jun 2022 05:01:01.846 * RDB age 7464 seconds
30878:M 01 Jun 2022 05:01:01.846 * RDB memory usage when created 2.50 Mb
30878:M 01 Jun 2022 05:01:01.846 * RDB has an AOF tail
30878:M 01 Jun 2022 05:01:01.846 * Reading the remaining AOF tail...
30878:M 01 Jun 2022 05:01:01.846 * DB loaded from append only file: 0.000 seconds
30878:M 01 Jun 2022 05:01:01.846 * Ready to accept connections
48593:C 01 Jun 2022 05:16:19.128 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
48593:C 01 Jun 2022 05:16:19.128 # Redis version=6.2.0, bits=64, commit=00000000, modified=0, pid=48593, just started
48593:C 01 Jun 2022 05:16:19.128 # Configuration loaded
48593:M 01 Jun 2022 05:16:19.247 * Increased maximum number of open files to 10032 (it was originally set to 1024).
48593:M 01 Jun 2022 05:16:19.247 * monotonic clock: POSIX clock_gettime
48593:M 01 Jun 2022 05:16:19.247 * Node configuration loaded, I'm 17ff68e0800d8aea279bddacf27c5d2d0cfb52a0
48593:M 01 Jun 2022 05:16:19.247 * Running mode=cluster, port=7005.
48593:M 01 Jun 2022 05:16:19.247 # WARNING: The TCP backlog setting of 511 cannot be enforced because /proc/sys/net/core/somaxconn is set to the lower value of 128.
48593:M 01 Jun 2022 05:16:19.247 # Server initialized
48593:M 01 Jun 2022 05:16:19.248 * Reading RDB preamble from AOF file...
48593:M 01 Jun 2022 05:16:19.248 * Loading RDB produced by version 6.2.0
48593:M 01 Jun 2022 05:16:19.248 * RDB age 8382 seconds
48593:M 01 Jun 2022 05:16:19.248 * RDB memory usage when created 2.50 Mb
48593:M 01 Jun 2022 05:16:19.248 * RDB has an AOF tail
48593:M 01 Jun 2022 05:16:19.248 * Reading the remaining AOF tail...
48593:M 01 Jun 2022 05:16:19.248 * DB loaded from append only file: 0.000 seconds
48593:M 01 Jun 2022 05:16:19.248 * Ready to accept connections

一点用都没。
发现7002有主没从。
那就加7005到7002。

改变想法。它没了，我就添加它。

 src/redis-cli --cluster add-node 192.168.1.135:7005 192.168.1.133:7002 --cluster-slave --cluster-master-id ba4f83ec20945617d089adf7db640a6052cb2fa5
>>> Adding node 192.168.1.135:7005 to cluster 192.168.1.133:7002
>>> Performing Cluster Check (using node 192.168.1.133:7002)
M: ba4f83ec20945617d089adf7db640a6052cb2fa5 192.168.1.133:7002
   slots:[5461-10922] (5462 slots) master
M: e1804de236f04b121ffff5bb5c5e661adfa50c66 192.168.1.135:7004
   slots:[10923-16383] (5461 slots) master
   1 additional replica(s)
S: 2f6d776193a48b4dd39c908a6f0cd7c99450746d 192.168.1.134:7001
   slots: (0 slots) slave
   replicates e1804de236f04b121ffff5bb5c5e661adfa50c66
S: 70d81f0d43f2bc131ad7356711c5b49d49d10507 192.168.1.133:7003
   slots: (0 slots) slave
   replicates 1731b6bbca92f945dd01cf589c8f0fbc5048205b
M: 1731b6bbca92f945dd01cf589c8f0fbc5048205b 192.168.1.134:7000
   slots:[0-5460] (5461 slots) master
   1 additional replica(s)
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
[WARNING] Node 192.168.1.135:7004 has slots in importing state 5798.
[WARNING] Node 192.168.1.134:7000 has slots in importing state 5798,12958,13593.
[WARNING] The following slots are open: 13593,12958,5798.
>>> Check slots coverage...
[OK] All 16384 slots covered.

重点来了

[ERR] WARNING] Node 192.168.1.135:7004 has slots in importing state 5798.
[WARNING] Node 192.168.1.134:7000 has slots in importing state 5798,12958,13593.
[WARNING] The following slots are open: 13593,12958,5798.

说明有几个slots正在输入。而我确定这不可能啊。那他就错了。
问题一解决方法
进入另外的集群机器。删除相关槽数据

[root@localhost redis]# src/redis-cli -h 192.168.1.134 -c -p 7000
192.168.1.134:7000> cluster setslot 13593 stable
OK
192.168.1.134:7000> cluster setslot 5798 stable
OK
192.168.1.134:7000> cluster setslot 12958 stable
OK

[root@localhost redis]# src/redis-cli -h 192.168.1.134 -c -p 7000
192.168.1.134:7000> cluster setslot 5798 stable
OK

再从新添加就好了

[root@localhost redis]# src/redis-cli --cluster add-node 192.168.1.135:7005 192.168.1.133:7002 --cluster-slave --cluster-master-id ba4f83ec20945617d089adf7db640a6052cb2fa5
>>> Adding node 192.168.1.135:7005 to cluster 192.168.1.133:7002
>>> Performing Cluster Check (using node 192.168.1.133:7002)
M: ba4f83ec20945617d089adf7db640a6052cb2fa5 192.168.1.133:7002
   slots:[5461-10922] (5462 slots) master
M: 1731b6bbca92f945dd01cf589c8f0fbc5048205b 192.168.1.134:7000
   slots:[0-5460] (5461 slots) master
   1 additional replica(s)
S: 70d81f0d43f2bc131ad7356711c5b49d49d10507 192.168.1.133:7003
   slots: (0 slots) slave
   replicates 1731b6bbca92f945dd01cf589c8f0fbc5048205b
S: 2f6d776193a48b4dd39c908a6f0cd7c99450746d 192.168.1.134:7001
   slots: (0 slots) slave
   replicates e1804de236f04b121ffff5bb5c5e661adfa50c66
M: e1804de236f04b121ffff5bb5c5e661adfa50c66 192.168.1.135:7004
   slots:[10923-16383] (5461 slots) master
   1 additional replica(s)
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.
>>> Send CLUSTER MEET to node 192.168.1.135:7005 to make it join the cluster.
Waiting for the cluster to join

>>> Configure node as replica of 192.168.1.133:7002.
[OK] New node added correctly.

查询就有了

192.168.1.133:7002> cluster nodes
b5fbed0a0bb1dbf44275f473ca09bdfd79f4d64a 192.168.1.135:7005@17005 slave ba4f83ec20945617d089adf7db640a6052cb2fa5 0 1654078383000 8 connected
1731b6bbca92f945dd01cf589c8f0fbc5048205b 192.168.1.134:7000@17000 master - 0 1654078383620 1 connected 0-5460
70d81f0d43f2bc131ad7356711c5b49d49d10507 192.168.1.133:7003@17003 slave 1731b6bbca92f945dd01cf589c8f0fbc5048205b 0 1654078382615 1 connected
2f6d776193a48b4dd39c908a6f0cd7c99450746d 192.168.1.134:7001@17001 slave e1804de236f04b121ffff5bb5c5e661adfa50c66 0 1654078381610 5 connected
e1804de236f04b121ffff5bb5c5e661adfa50c66 192.168.1.135:7004@17004 master - 0 1654078382000 5 connected 10923-16383
ba4f83ec20945617d089adf7db640a6052cb2fa5 192.168.1.133:7002@17002 myself,master - 0 1654078382000 8 connected 5461-10922