Partitioning and Consistent Hashing

Success of any distributed storage system is majorly dependent on the data partitioning and replication scheme that it uses. For a distributed system to be highly available and reliable, it needs to function business as usual even in the case of node loss.

With ever growing data in terms of volume, all of it cannot fit in a single node and hence came the distributed storage systems with horizontal scalability. Data partitioning is a mechanism in which data is partitioned and stored in multiple nodes. In order to address the missing nodes(servers), data partitions are required to be replicated in more than one node, which could vary from any where starting with racks to different geographical regions and anything in between. This way of storing same copy of data partition is called Data replication. Again data replication enables fault tolerance or partition tolerance.

There are quite a few partitioning schemes that allocates different partitions to different servers, like simple hash based partitioning and range based partitioning.

Simple hash based partitioning : General procedure for hash based partitioning is for a given data element (a row or anything that is taken as an entity) , come up with a key and applying a hash function to map it to a number. Then do a modulo on hash function value and the number of servers. The resulting value would be in the range 1 to the number of servers added

(hash(key)%number of servers) = a number(in the range of servers)

This is the base idea of HashTables as well. And yes of course, collisions can happen if the hash function generates same hash for two different keys. The robustness of this scheme largely depends on the chosen hash function and the randomization of the key.

Problem: Major problem with this simple hash based partitioning is that when nodes come down, the data on that node needs to be moved around to other nodes in the cluster. Also, the count of nodes changes when nodes scale up or scale down and hence the re mapping of all keys is required, which involves huge data movement. Needless to say it is most expensive operation to handle without making availability trait to suffer.

Range based partitioning : In this partitioning scheme, certain ranges are allocated to nodes, and if hash or any predefined function on the chosen key falls into a certain range, then corresponding server stores the data.

Problem: In this partitioning technique there is possibility of data being skewed on some nodes vs the others. Again, the success of this depends on how well the function to decide the range is selected.It also exhibits the need to rebalance or re distribute data in the cluster in case of adding nodes and decommissioning nodes.

Taking good parts from both hash based and rage based partitioning, consistent hashing alleviates the need for aggressive rebalancing of data. Consistent hashing uses a concept called Virtual nodes or Vnodes.

In this partitioning technique, each physical node in the cluster is assigned with a set of Vnodes. Instead of assigning a token or consecutive range of tokens to a physical node, each Vnode of physical node handles smaller hash ranges. Each Vnode is replicated once, to account for fault tolerance.

The main advantage of consistent hashing is that, Vnodes help to distribute load across the cluster. When a node goes down or gets added to existing cluster, only small amount of data needs to be re distributed as each Vnode handles smaller range of tokens unlike physical nodes without Vnodes.

As an example, distributed data store Cassandra uses consistent hashing for data partitioning and replication.

References:

Related posts:

Agora deslizo pelas músicas de seda negra de um cabaret, pois é onde encontro meu conforto. No cabaret e no jazz, este quanto mais sujo, melhor. Não apenas o cenário que sempre, de forma sombria…

If you wonder if your child has dyslexia, here is an overview of this condition. Dyslexia is a condition where someone may struggle with reading fluently. One sign of dyslexia is the inability to…

I read and meditate on her every morning. Biographical accounts, nautical recollecitons, economic analysis. I try to logically pry my mind off of Havana. A whispering dream. The maverick sailors, the…