While the resharding is in progress you should be able to see your example program running unaffected. You can stop and restart it multiple times during the resharding if you want.
At the end of the resharding, you can test the health of the cluster with the following command:
./redis-trib.rb check 127.0.0.1:7000
All the slots will be covered as usually, but this time the master at 127.0.0.1:7000 will have more hash slots, something around 6461.
*Scripting a resharding operation
Reshardings can be performed automatically without the need to manually enter the parameters in an interactive way. This is possible using a command line like the following:
./redis-trib.rb reshard: --from --to --slots --yes
This allows to build some automatism if you are likely to reshard often, however currently there is no way forredis-trib to automatically rebalance the cluster checking the distribution of keys across the cluster nodes and intelligently moving slots as needed. This feature will be added in the future.
*A more interesting example application
The example application we wrote early is not very good. It writes to the cluster in a simple way without even checking if what was written is the right thing.
From our point of view the cluster receiving the writes could just always write the keyfoo to42 to every operation, and we would not notice at all.
So in the redis-rb-cluster repository, there is a more interesting application that is calledconsistency-test.rb. It uses a set of counters, by default 1000, and sendsINCR commands in order to increment the counters.
However instead of just writing, the application does two additional things:
When a counter is updated using INCR, the application remembers the write. It also reads a random counter before every write, and check if the value is what we expected it to be, comparing it with the value it has in memory.
What this means is that this application is a simple consistency checker, and is able to tell you if the cluster lost some write, or if it accepted a write that we did not received acknowledgment for. In the first case we'll see a counter having a value that is smaller than the one we remember, while in the second case the value will be greater.
Running the consistency-test application produces a line of output every second:
$ ruby consistency-test.rb 925 R (0 err) | 925 W (0 err) | 5030 R (0 err) | 5030 W (0 err) | 9261 R (0 err) | 9261 W (0 err) | 13517 R (0 err) | 13517 W (0 err) | 17780 R (0 err) | 17780 W (0 err) | 22025 R (0 err) | 22025 W (0 err) | 25818 R (0 err) | 25818 W (0 err) |
The line shows the number of Reads and Writes performed, and the number of errors (query not accepted because of errors since the system was not available).
If some inconsistency is found, new lines are added to the output. This is what happens, for example, if I reset a counter manually while the program is running:
$ redis 127.0.0.1:7000> set key_217 0 OK (in the other tab I see...) 94774 R (0 err) | 94774 W (0 err) | 98821 R (0 err) | 98821 W (0 err) | 102886 R (0 err) | 102886 W (0 err) | 114 lost | 107046 R (0 err) | 107046 W (0 err) | 114 lost |
When I set the counter to 0 the real value was 144, so the program reports 144 lost writes (INCR commands that are not remembered by the cluster).
This program is much more interesting as a test case, so we'll use it to test the Redis Cluster failover.
*Testing the failover
Note: during this test, you should take a tab open with the consistency test application running.
In order t