Levels

Each level teaches one idea. You get a system that is about to fail. Figure out why, change the design, and keep users happy until the run ends. Every level can be beaten, and each one ends with a reference design and an explanation.

1. 1
   One server is not enoughCapacity

   Your startup launched. 600 people per second are hitting a single API server.

   Playreset level
2. 2
   Don't put all your eggs in one basketReplicas & health checks

   Servers crash: out of memory, a bad disk, a kernel panic. Not if, but when.

   pass level 1
3. 3
   The database is always the bottleneckCaching & read replicas

   Your API tier scales easily: just add servers. The database does not. It has a fixed connection pool and every read and write goes through it.

   pass level 2
4. 4
   Your cache is lying to youCold caches & headroom

   Caches make systems look much stronger than they are. A 90% hit ratio means the database only sees 1 in 10 reads.

   pass level 3
5. 5
   Retries can make things worseTimeouts, retries & backoff

   Your team made the API 'resilient': every failed database call is retried 3 times, immediately.

   pass level 4
6. 6
   Not everything is urgentQueues & async work

   Half your traffic is writes: orders being placed. Writes are expensive for the database.

   pass level 5
7. 7
   The network is not reliablePartitions & failing fast

   Two API servers talk to the database over the network. Networks break in strange ways: packets just stop arriving, and nobody gets an error, only silence.

   pass level 6
8. 8
   One customer is 60% of your trafficSharding & hot keys

   Writes outgrew one database, so the data is split across four shards by key.

   pass level 7
9. 9
   Everyone at 9:00Thundering herds

   A scheduled job sends notifications every minute by calling your API directly, the same API that serves users.

   pass level 8
10. 10
    On-call nightEverything together

    Final exam. You are on call. Chaos is set to Brutal: several different failures, sometimes overlapping, with misleading log noise.

    pass level 9