Manual failovers are supported by Redis Cluster using the CLUSTER FAILOVER command, that must be executed in one of the slaves of the master you want to failover.
Manual failovers are special and are safer compared to failovers resulting from actual master failures, since they occur in a way that avoid data loss in the process, by switching clients from the original master to the new master only when the system is sure that the new master processed all the replication stream from the old one.
This is what you see in the slave log when you perform a manual failover:
# Manual failover user request accepted. # Received replication offset for paused master manual failover: 347540 # All master replication stream processed, manual failover can start. # Start of election delayed for 0 milliseconds (rank #0, offset 347540). # Starting a failover election for epoch 7545. # Failover election won: I'm the new master.
Basically clients connected to the master we are failing over are stopped. At the same time the master sends its replication offset to the slave, that waits to reach the offset on its side. When the replication offset is reached, the failover starts, and the old master is informed about the configuration switch. When the clients are unblocked on the old master, they are redirected to the new master.
*Adding a new node
Adding a new node is basically the process of adding an empty node and then moving some data into it, in case it is a new master, or telling it to setup as a replica of a known node, in case it is a slave.
We'll show both, starting with the addition of a new master instance.
In both cases the first step to perform is adding an empty node.
This is as simple as to start a new node in port 7006 (we already used from 7000 to 7005 for our existing 6 nodes) with the same configuration used for the other nodes, except for the port number, so what you should do in order to conform with the setup we used for the previous nodes:
Create a new tab in your terminal application. Enter the cluster-test directory. Create a directory named 7006. Create a redis.conf file inside, similar to the one used for the other nodes but using 7006 as port number.Finally start the server with ../redis-server ./redis.conf
At this point the server should be running.
Now we can use redis-trib as usually in order to add the node to the existing cluster.
./redis-trib.rb add-node 127.0.0.1:7006 127.0.0.1:7000
As you can see I used the addnode command specifying the address of the new node as first argument, and the address of a random existing node in the cluster as second argument.
In practical terms redis-trib here did very little to help us, it just sent a CLUSTER MEET message to the node, something that is also possible to accomplish manually. However redis-trib also checks the state of the cluster before to operate, so it is a good idea to perform cluster operations always via redis-trib even when you know how the internals work.
Now we can connect to the new node to see if it really joined the cluster:
redis 127.0.0.1:7006> cluster nodes 3e3a6cb0d9a9a87168e266b0a0b24026c0aae3f0 127.0.0.1:7001 master - 0 1385543178575 0 connected 5960-10921 3fc783611028b1707fd65345e763befb36454d73 127.0.0.1:7004 slave 3e3a6cb0d9a9a87168e266b0a0b24026c0aae3f0 0 1385543179583 0 connected f093c80dde814da99c5cf72a7dd01590792b783b :0 myself,maste