Identiy, Users and Access: The holey trinity


Identity, Users and Access: The holey trinity

For some time I have been meaning to capture a snapshot of some of the key design considerations for the identity and account structure which emerged from quite a few years of requirements spanning Ad-tech, dating and multi account systems built across multiple startups.

The reasoning for sharing this model is in the hope that some of the lessons can be useful for others faced with similar challenges. If nothing else the model, as an emergent real world solution has some quirks not common in many or all public solutions in this space.

Requirements at scale of multiple companies

  • In the ad tech space there is an interesting precedent designed to protect the privacy of users. Publishers and advertisers each see a different 'user id'. This is the ID that is stored in the impression and click logs which drive the entire ad industry. One of the startups explicitly needed support for the concept of scoped anonymised user identity as this was an artefact of the underlying data. We actually scoped this up to a higher level requirement for all systems and it worked nicely to avoid ever exposing the internal ID of a user.

  • Separation of personal, an identity and user. This allows personally identifiable information to be separated from a 'user'. The concept of a user is very overloaded and is usually an account. In most real world systems you see there is a need to be able to have multiple people log in to the same account. This necessitates separation of the identity from the user[name].
  • Users will probably want to log in with lots of different IDP and other sources. It's worth noting too that it's helpful to be able to have multiple grants stored for the same user.
  • It's the identity, not the person or the username to which logic needs to be applied.
  • At the time of writing our access system used explicit scoped grants to gift access. These scopes were effectively paths of entities. The paths for a user access were consistently named also with our tasks and other frameworks. It would have been nice to revisit this and re-scope access to identity with hindsight.
  • Person Name was of particular interest as it's interesting to be able to store alias easily. Notably most systems aren't built with this up front. It is a very common use case however in production systems.



The model evolved over time and as such went through a number of iterations. There are a few quirks that warrant further attention. User access was one such quirt, arguably it should be associated with the identity rather than the user concept. It ended up living in user because at the time of creation, identity and access modules were in separate packages. Access should have been separated into another module.

Looking back another thing that seemed odd is that we didn't have even a group concept. The reason for this is that the 'thing' would often be a group. The access did have a collection of types of grant which were originally based on unix style read / write / x bits. These concepts expanded over time to contain about 7 different types of permission however they didn't explode out of control.

Startup ENG Rules Series. 1. Storage. Prefer Insertions to Updates

box doesn't fit in circles















Over the last decade I have worked at a number of very large and very small companies, the smallest being just two people and the largest having over one hundred thousand people. On day one a startup faces many challenges however the most important one is usually survival. The first couple of months are critical while going from 'zero to one' as the saying goes. The top priority has to be to secure customers and launch an initial offering, these days often referred to as an Minimum Viable Product. In this mode engineers need to build just enough to make the product work. Like Occams Razor all non essential work should likely be avoided and there is little room for high principles in the search for results.

One things that changes as a startup moves into iterating rather than prototyping is how to survive the first big customer. The focus of the team often has to shift from optimizing for a viable solution to consider aspects of stability, reliability and correctness. In some sectors these are more or less important compared to rapidly reaching a position where feedback can be sought to verify product hypothesis.

In this series of posts I am going to share a collection of 'rules' which emerged through a number of projects including the rationale, signals and counter cases.

Note: these rules are based on experience from startups and may not reflect common practices in larger companies. These insights are shared purely based on experiences from:

  • SiteMorph: A SEO / SEM marking tool for SMB.
  • ClickDateLove.com (Muster): A dating site employing basic ML approaches to create better profiles.
  • Shomei / Futreshare: Ad attribution heuristic modelling for advertisers with billions of ad impressions.
  • Upgrade Digital: hospitality booking platform build for developers with one of the fastest build times for web developers available in the world at the time.

 The objective of these rules was to have standard solutions to everyday questions based on real world lessons. Having de-facto solutions to everyday problems meant that development could go faster. Going faster for a startup means less cost, faster iteration and more feedback. Some of the rules may seem to contradict this when they add overhead. The point here is that the solutions were born out of necessity. This necessity drove iteration to a viable solution.

Rule 1. Storage, prefer insertions to updates


When a data attribute of an entity may be written or updated by a number of writers, prefer refactoring that at attribute into a separate concept and inserting in into a different store rather than updating a field on an existing entity.

  •  Payment authorization code for a payment
  • Approval for a change where multiple people can approve
  • Any transaction sensitive attribute which could be the source of a race condition.


Consider a hotel booking for a single stay this can be expressed in normal form along the lines of:

  • Hotel Booking
    • user : who books the stay
    • checkin : date of arrival
    • rooms : ... details of the required rooms
    • total cost: sum of all room night rates and fees.
    • payment request: payment transaction token used to initiate the transaction.
    • payment confirmation: payment completion token from transaction processor.
    • payment cancellation: the cancellation token passed by the transaction processor.

This seems pretty reasonable and has all of the fields associated with the booking however without good locking of the entity type multiple actors are able to update the fields leading to a lost update race condition. Locking isn't such a bad thing you may argue however underlying locking semantics typically lead to centralization as decentralized consistency isn't offered by many storage engines and CAP theorem comes into play. Rather than update attributes of the existing booking, one typically safe solution well aligned with eventual consistency offered by many storage engines is to always insert. To achieve this the entities need to be separated like so:

  • Hotel booking
    • user
    • checkin
    • rooms
    • total cost
  • Payment request
    • hotel booking reference
    • payment request token
  • Payment confirmation
    • payment request reference
    • payment confirmation token
  • Payment cancellation
    • payment confirmation reference
    • payment cancellation token


  1. Avoiding locks helps us to scale better. Many storage engines only support table level locks which can be a significant issue in online transaction processing systems. One payment provider I have worked with had median API response times in the 1000+ms range. Even the best available are often still in the 200ms range. Effectively this means if you hold a lock to update your booking, or payment table, you can only process ~5 transactions per second. Always inserting typically has O(1) performance semantics and is typically only limited in performance by disk / network speed.
  2. Avoiding lock release starvation is a significant gain. In the world of scaled data centres it's only a matter of time before one of your service is going to crash during a transaction. The law of large numbers says that as you have more services you are likely to start to observe more instance crashes. With a 2x9s 99.5% you still have 12960 seconds every month of downtime per instance to contend with. Even using advanced monitoring you can't avoid some crashes at scale. Given that it's essential to plan for them. When a process crashes, most distributing locking solutions will have to wait for automated timeout of the lock. Eliding this problem by locking is a significant win in degraded situations.
  3. Minimise the window for issues. Recovery is always required but writing updates with O(1) insert semantics dramatically narrows the window for lost writes. For our storage system at the time we were seeing insertion times in nanoseconds for first disk flush. At this point we only saw one crash during insert per year. We build a recovery task for that too. Keep an eye out for the future rule on self correction.
  4. Minimise your entity storage reliance on technology specific sophisticated locking e.g. relational database locks.


  • For upgrade digital one of our key value propositions is that our platform included correction of booking state and payment processing providers. One of the hotel chains we worked with regularly had rooms without payment and payment without rooms!
    • Some payment providers used had delays in correction of up to 24 hours in production so we had to recover elegantly. This might lead to retrying a transaction that had previously timed out only to see it later succeeded so we needed to keep all request initialisation vectors.
    • Hotel room booking systems often allow manual overrides for room allocations as well as overbooking as a standard practice. This could mean that the actual product wasn't available for extended periods of time.
  • For general payment systems it's good to practice to expect delays in callbacks and generally avoid overwriting fields as race conditions and replay are regular occurences.
  • For hospitality the Upgrade Digital platform provided a consistent RESTful API across multiple Micros Opera versions and a number of payment processors. Our approach to play / replay / check async task execution automatically repaired numerous issues on either side of the platform automatically meaning we could sleep at night. For a small oncall team supporting bookings across 120 countries this is a must!

Counter cases

Despite the general practice of always inserting there are notable counter cases where we did use basic locking functionality with a 'test and set' semantic:

  • In our task scheduling the library used a task claim, compatible with AWS SQS to claim async work. This claim required a test and set style storage engine which was easy to achieve with SQL and some no-sql storage engines like Dynamo.
  • Critical sections of code where exactly once semantics are required.