May 30, 2017

Consistency with Cthulhu

Understanding Consistency Models, A Practical Guide

Consistency is complicated. While that is likely not the opening line you were hoping for, it’s best to set the proper expectations.

Oftentimes when dealing with consistency challenges, developers are forced to choose between two primary types of consistency: strong (data is not available until total replication) and eventual (data may not be available on replications immediately). Forcing developers to choose often results in slow or inconsistent systems, with sometimes weird nuances. For example, with eventual consistency, if you are using a social media service, comments may appear out of order until the replication is complete, which may take minutes, depending on the load and services consumed. In conversations, order is important, or you’ll end up confused and the user experience is degraded due to poor service design. Another common challenge is fault tolerance replication, which all but forces developers to utilise strong consistency, so all writes are read after replication is guaranteed, meaning your system has the proper order of events.

When explaining consistency models to developers or other engineers, you can often see the look of dread on their face, or the wheels in their head turning as they try to figure out which consistency model is the best model for their application design. Many times, I’ve had to have discussions with coworkers and other teams about why a consistency model was chosen, and how that will impact their downstream consumption as it relates to our internal design decisions. These conversations are never fun, and they almost always end up with my apologies for strange system behaviour when trying to overcome the limitations of the Laws of Physics:

When looking at the new consistency models for Azure Cosmos DB, I got very excited. I ended up having a 45 minute discussion at //build this year with one of the engineers, to have him explain to me how it works in detail, and some of the challenges they came across. I was very excited because sometimes, for the best experiences, you can’t truly use one or the other; the experience wouldn’t be as intended. Today, I wanted to write a practical reference about using the varying consistency models Cosmos DB utilises as it relates to battling the Elder God, Cthulhu. To make sure our battle is fair, we have summoned the Black Pharaoh, Nyarlathotep. Nyarlathotep, as the Black Pharaoh, will monitor the battle and will decide on the victor. Cthulhu and Nyarlathotep are references to HP Lovecraft’s fictional universe, where cosmic horrors exist to terrorise the lowly populations. I chose to work with a Dungeons & Dragons style through Lovecraft’s universe as it is a great way to apply complicated computational science topics in a way that anyone can understand; we can use our hands to demonstrate and visually show what would happen at certain points in the topic.

As part of these references, we will use 5 items: Cthulhu (with 50 health), your character (with 20 health), a 20-sided die (D20) for you, a D20 for Cthulhu, and Nyarlathotep. You and Cthulhu are two applications, you use your own dice, as it is representative of differing business logic within your applications. Nyarlathotep is representative of Cosmos DB, as that is where you will be tracking the health of each entity.

You are traveling across the Forgotten Shores, in search of the remains of the Dreaded Man, a Man O’ War ship lost centuries ago. It contains a map to the Epirus Bow. The Epirus Bow is a bow immense power, and with it, you may conquer the Lurker at the Threshold and free your lands of the insanity bestowed upon you. As you reach the end of the peninsula, the sea begins to foam around you, and lo, the High Priest of the Great Old Ones rises from the deep. As Cthulhu begins to engage with you, you summon Nyarlathotep, the Black Pharaoh, to reside over the battle, keeping tallies, and updating the Official Records during/after the battle. Let us begin.

Strong Consistency

Cthulhu gazes upon you, seeing your mortality. As your encounter begins, Nyarlathotep promises all damage will be resolved before the other entity rolls. He rolls his D20; a seven is rolled. Nyarlathotep acknowledges the wound, marks it in his tally. You see Nyarlathotep’s tally mark, and wounded at 13 health, roll. Your die falls on the floor, but you roll a 12. You tell Nyarlathotep to mark a 12, however, Cthulhu sees your fallen die, and asks for a reroll. You roll again, this time rolling a three. You tell Nyarlathotep to mark a three, and he does so. Cthulhu reads the three, and shrugs off your feeble blows. Cthulhu rolls, rolling a one. “Impossible!”, says the Horror, and rolls again, before Nyarlathotep has a chance to read the original roll! He rolls a 20. Nyarlathotep updates his tally. You read Nyarlathotep’s tally, and you die.

This is an example of strong consistency. Damage is not assigned to either entity until Nyarlathotep has properly tallied in his book and updated the Official Records during the encounter.

Bounded Staleness

Cthulhu gazes upon you, seeing your mortality. As your encounter begins, Nyarlathotep strikes a bargain with you, promising his tallies will never be more than one roll behind; so if Cthulhu rolls, but rerolls multiple times, Nyarlathotep promises you will be only one roll behind at any given time. Cthulhu, as a Cosmic Horror, strikes at you, rolling a seven. Nyarlathotep acknowledges the wound, and marks it in his tally. You read the tally, and wounded at 13 health, roll. Your die falls on the floor, but you roll a 12. Nyarlathotep marks 12 in his tally. Cthulhu goes to read the tally, but you fight with honor, so you reroll a three and have the tally updated. Cthulhu sees Nyarlathotep’s updated tally, and strikes again! He rolls, rolling a one! “Impossible! I shall strike again!” Cthulhu rerolls, rolling another one! Nyarlathotep has marked both ones, and as you go to read from the tally, Cthulhu rolls again, this time a 15! Nyarlathotep updates his tally as you see it, but you only see Nyarlathotep’s second mark of one. As you go to roll, Nyarlathotep audits his tally, and shows you the updated mark. Seeing your updated damage taken, you die.

This is an example of bounded staleness consistency. Both entities are in agreement through Nyarlathotep that damage tallies will never be out of date by more than one roll. The Official Records are never kept more than one roll behind the encounter’s current damage assessment.


Cthulhu gazes upon you, seeing your mortality. As your encounter begins, Nyarlathotep decides a tokened promise will be shared. For this encounter, you will each need to share a token, so that all damage is consistent. For Nyarlathotep’s records, only the final outcome of the battle will be recorded, no damage is permanent in the archives until the battle is over. Cthulhu asks for a token from Nyarlathotep, and with his token in hand, strikes at you, rolling a seven. Nyarlathotep marks seven in his tally and hands you the token. You read the tally, and while your die falls on the floor, you roll a 12. With your roll complete, you return the token to Nyarlathotep. Cthulhu takes the token, and rolls a 15! Nyarlathotep accepts the token, and marks in his tally. You accept the token, and you die.

This is an example of session consistency. Nyarlathotep generates a shared token for your encounter, and while all damage is resolved during the encounter, he does not archive the encounter into the Official Records until the encounter is over and the token has been released.

Consistent Prefix

Cthulhu gazes upon you, seeing your mortality. As your encounter begins, Nyarlathotep promises that while multiple rolls may occur from either entity, you will always see the damage caused in order, regardless of how many rolls occur. Cthulhu rolls, rolling a seven. Nyarlathotep updates his tally. You read the tally, and wounded, roll a three, this time landing properly. Nyarlathotep updates his tally, and Cthulhu rolls. One! The Horror rolls again, another one! Cthulhu, angry at Luck, rolls again, rolling a perfect 20! Nyarlathotep has been keeping tally, but you read the tally after the first roll. While you have only taken one damage, you fear the Fates are playing jokes on you, so you read Nyarlathotep’s tally again. You see Cthulhu has struck you three times, two strikes for one damage each, and then his roll for a perfect 20! You die.

This is an example of consistent prefix consistency. As an entity performs a roll, that is updated in Nyarlathotep’s tally book. Each roll is updated in the Official Records, but Nyarlathotep isn’t keeping the records up to date at any given time. However, should someone read the Official Records during the battle, they may or may not see all three of Cthulhu’s rolls, they may see only one or two of three rolls. However, Nyarlathotep has promised the Official Records will never contain out of order damage resolutions, as it’s unfair to readers, giving them a false accounting of the battle held.

Eventual Consistency

Cthulhu gazes upon you, seeing your mortality. As your encounter begins, Nyarlathotep tells you his tally may be out of date at any given roll, and to be cautious when approaching battle. Cthulhu strikes at you, rolling a seven. Nyarlathotep marks the damage in his tally. You read his tally, and wounded, roll a 12! Feeling confident, you watch as Nyarlathotep marks the tally in his book. As you return your focus to the battle, you trip on a rock, causing you to become dazed! Cthulhu shrugs off your pathetic attempt at wounding the High Priest, and rolls a two. Angered, he rolls again, taking advantage of your dazed confusion; rolling a three. As the Cosmic Horror emits a noise too despicable to record, he rolls a third time! A 19 has been cast. While Nyarlathotep has been keeping tallies, you are so dazed you only see the first one. Taking two damage, you make an attempt to roll, rolling a five. Telling Nyarlathotep your roll, you suddenly see Cthulhu’s last roll, and you die, taking 19 damage.

This is an example of eventual consistency. Nyarlathotep has promised the Official Records will eventually converge with his tally, but does not promise when. He promises they will eventually get updated, but he makes no commitment, in his capacity as the Black Pharaoh, as to when the records will be up-to-date, thus potentially leaving readers wondering if you actually defeated the High Priest!

Parting Thoughts

While these examples are simplistic, they give an example of ways that your applications can utilise the varying forms of consistency modeling within Cosmos DB. You can see examples of what would happen with record resolution, life or death situations, and what future readers could see of the historical records of your battle with the Elder Gods. There is no right consistency model for all situations, and as such you should review which consistency model will best suit your use cases.