System Design - Fundamentals

This is Chapter 1 of the System Design series.

Architecture reviews and design discussions keep coming back to the same vocabulary regardless of what’s being built. I keep this list so I’m reasoning from consistent definitions each time, not re-deriving the same trade-offs from scratch.

Core concepts

1. Scalability

• Vertical scaling (bigger machine) buys time but hits a ceiling fast and creates a single point of failure.
• Horizontal scaling (more machines) is the default for anything expected to grow past a single box, at the cost of needing to handle state, coordination, and partial failure across nodes.

2. Availability

Usually expressed in nines:

Each additional nine costs disproportionately more in redundancy, automation, and operational discipline. Match the target to what the business actually needs - five nines on an internal reporting tool is wasted spend.

3. Latency vs throughput

• Latency is how long one request takes.
• Throughput is how many requests the system handles per unit of time.

They trade off: batching improves throughput by adding latency to individual requests. A system tuned for one is rarely optimal for the other.

4. Consistency models

• Strong consistency means every read sees the latest write, which requires coordination and costs latency.
• Eventual consistency allows reads to lag behind for a window, in exchange for availability and lower latency.

Most systems don’t need strong consistency everywhere - decide it per data type, not per system.

5. CAP theorem and PACELC

• CAP: under a network partition (P), a system chooses between consistency (C) and availability (A); it cannot have both.
• PACELC extends this to the normal case: even without a partition, there’s a trade-off between latency (L) and consistency (C).

CAP gets cited more than it’s useful day to day - PACELC’s latency/consistency trade-off is the one that shows up in almost every real design decision.

6. Idempotency

• An operation that produces the same result no matter how many times it’s applied.

Required for safe retries - a payment API without idempotency keys will double-charge on a network timeout where the client retries a request the server actually processed.

7. Fault tolerance

• The system keeps serving correct results when a component fails.

Built from redundancy (no single point of failure), health checks, timeouts, and graceful degradation - not from hoping the failure doesn’t happen.

Building blocks

1. Load balancers

Distribute requests across instances:

2. Caches

Sits between the application and a slower data source. The two decisions that matter:

• Eviction policy: which entry gets removed when the cache is full.
• Write strategy: how writes flow between the cache and the backing store.

Cache invalidation is the hard part in practice, not lookup speed - stale data from a forgotten invalidation path causes more incidents than the cache ever saves in latency.

3. Databases

• SQL stores structured, relational data with schemas, joins, and ACID transactions.
• NoSQL covers a family of non-relational stores, each optimised for a different access pattern.

4. Message queues

Kafka, RabbitMQ, SQS, and Pub/Sub decouple producers from consumers, absorb load spikes, and let consumers fail and recover without losing work.

• At-least-once delivery is simpler and pairs with idempotent consumers.
• Exactly-once delivery adds real coordination complexity for a guarantee most systems don’t strictly need.

5. API gateways and reverse proxies

In front of backend services, so individual services don’t each re-implement them:

• Centralise routing
• Authentication
• Rate limiting
• TLS termination

6. CDNs

The right tool for anything that doesn’t change per-request:

• Cache static assets at edge locations
• Cut latency and offload origin traffic

7. Object storage

Large, immutable blobs that don’t belong in a relational database:

• Images and video
• Backups
• Logs

8. Vector databases and LLM gateways

• Vector databases store embeddings for similarity search, the backbone of retrieval-augmented generation.
• LLM gateways centralise model routing, rate limiting, and cost tracking across providers the same way an API gateway does for internal services.

Architecture patterns

1. Sharding and partitioning

Splits data across nodes so no single node holds the whole dataset:

• Consistent hashing on a key is the standard distribution mechanism
• The shard key is the one decision in a sharded system that’s expensive to change later - get it wrong and every cross-shard query becomes a fan-out. Spend the most time here.

2. Replication

Copies data across nodes for redundancy and read scaling:

• Leader-follower replication has one node accept writes and others replicate from it - simple, but the leader is a bottleneck and a single point of failure for writes.
• Multi-leader replication accepts writes on multiple nodes, trading that bottleneck for conflict resolution complexity.

3. Scaling reads vs writes

Read and write scaling are different problems:

• Read-heavy systems scale with replicas and caching.
• Write-heavy systems scale with sharding, since replication doesn’t help when every node still has to process every write.

4. Event-driven architecture and pub-sub

Decouples producers from consumers through events rather than direct calls:

• A producer doesn’t need to know who’s listening
• New consumers can be added without touching the producer

5. Microservices

Trades a simpler deployment and debugging story for independent scaling and deployment per service:

• Network calls replace function calls
• Distributed-systems failure modes replace in-process ones
• The trade-off only pays off once team size and deployment cadence justify the operational overhead

6. Multi-region architecture

Addresses latency and disaster recovery:

• Latency - serve users from the nearest region
• Disaster recovery - survive a regional outage

Also reintroduces the CAP trade-off at a larger scale - cross-region writes either wait for consensus or accept eventual consistency.

Trade-offs

What’s different in 2026

• AI components (embeddings, vector search, RAG) show up in mainstream designs the same way caching and queues do - no longer a separate track.
• Cost is a first-class constraint, not an afterthought - a design that burns budget unnecessarily is a weaker answer than one that’s cost-aware from the start.
• Real-time is the default expectation rather than a special requirement to call out.
• Multi-region reliability, once reserved for a handful of global platforms, is now assumed for anything with a meaningful user base.

How I work through a design problem

• Clarify requirements - functional scope, scale, and the constraints that actually shape the design (latency targets, consistency needs, budget)
• Estimate scale - rough numbers for traffic, storage, and bandwidth using the anchor points above
• Define the API - the contract the rest of the design has to support
• Sketch the high-level design - the major components and how data flows between them
• Deep-dive - pick the one or two components doing the most interesting work and go deep
• Address bottlenecks and failure modes - what breaks first under load, and what happens when it does

The order matters more than the content of any single step - skipping straight to a deep-dive before the scale is estimated produces a design that’s solving the wrong problem.

Notes