- When setting up a cluster, the most important number to worry about is ring_creation_size. This number, at the moment, cannot be changed once the cluster is up and running. Therefore, you need to plan ahead and pick a proper number for your cluster. One node should preferably handle at least 10 partitions. For example, if you intend to run a 5-node cluster right now, and foresee it grows to 10 nodes later, 128, and 256 are good values.
- Just like other database systems out there, file descriptor limit is a critical factor in Riak. The conservative figure is about 20 times the number of partitions (which is the same as ring_creation_size). So if you choose to have 128 partitions, you need to at least set open file limit (ulimit -n) to 2560. This should be done with pam_limits.
- Always prefer noatime mount option, unless you really need to know last access time.
- Again, prefer SSD for low latency disk access. If you are running on EC2, try not to use its Elastic Block Store.
- Bitcask storage backend requires that there is enough memory to store all keys. LevelDB introduces more latency when more levels are used. To reduce latency with LevelDB, add more nodes to the cluster. Secondary indexes only work with LevelDB. Secondary indexes are slow, very slow, so try to favor direct access with keys, or usage of another system to store these indexes.
- A Riak node does not need to leave a cluster in order to conduct repair work. Just do riak stop, and start repairing it. When it's ready, riak start. Try to wait for the cluster to converge before working on another node.
- Lastly, the best thing about Riak is its fault tolerance, true to the spirit of Erland. The cluster will eventually self heal, and converge, even if it seems slow at times.
PS: There is a good technical presentation given by the guys at Kiip about their use of Riak at http://basho.com/blog/technical/2012/05/25/Scaling-Riak-At-Kiip/. The presentation discusses good tips and pain points in a real world deployment.