Skip to content

System Design

System design turns requirements into a system that can meet explicit scale, reliability, security, and cost targets. The goal is not to name the most components; it is to expose assumptions and choose defensible trade-offs.

A Repeatable Process

1. Clarify the Problem

Define:

  • users, workflows, and business invariants;
  • inputs, outputs, APIs, and data lifecycle;
  • expected traffic, object sizes, and growth;
  • latency, availability, durability, and recovery objectives;
  • security, privacy, compliance, geography, and budget;
  • what is deliberately out of scope.

Separate hard constraints from preferences. Ask what failure means to the user.

2. Estimate the Order of Magnitude

Use transparent approximations rather than false precision.

average requests/s = requests/day ÷ 86,400
peak requests/s    = average × peak factor
storage/year       = writes/s × bytes/write × 31,536,000
bandwidth          = requests/s × bytes/request

Include replication, indexes, metadata, retention, and headroom. Estimates decide which problems matter; measurement replaces them later.

3. Model Data and Interfaces

Identify entities, identifiers, relationships, invariants, access patterns, retention, and ownership. Choose storage from those needs rather than from popularity.

Define contracts early enough to expose boundaries. Include errors, pagination, idempotency, authorization, and compatibility. See API Design and Databases.

4. Draw the Smallest Working Design

Start with clients, one application boundary, the primary data store, and essential external systems. Add caches, queues, replicas, partitions, gateways, and services only for a named bottleneck or isolation need.

Trace at least:

  • a normal read and write;
  • concurrent updates;
  • a dependency timeout;
  • overload and back-pressure;
  • deployment and rollback;
  • data loss and recovery.

5. Find Bottlenecks and Failure Modes

For every critical dependency ask:

  • What if it is slow, unavailable, stale, or returns malformed data?
  • Is there a deadline and bounded resource use?
  • Can the operation be retried safely?
  • What queues, and is the queue bounded?
  • What is the blast radius?
  • How is the condition detected and repaired?

6. Validate and Evolve

Load-test uncertain capacity assumptions, inject representative failures, test restore procedures, and observe production service-level indicators. Record the decision and its revisit trigger.

Scaling Toolkit

Vertical and Horizontal Scaling

Larger machines are operationally simple but have ceilings and larger failure domains. More instances improve capacity and replacement but require stateless request handling or explicit state coordination.

Load Balancing

Load balancers distribute requests and remove unhealthy targets. Common policies include round robin, least connections, weighted routing, and consistent hashing. Health checks should represent the instance's ability to serve traffic without creating synchronized load.

Keep session state outside individual instances when feasible. If affinity is necessary, define what happens when the selected instance fails.

Caching

Cache only after identifying an expensive, repeated access path.

For each cache define:

  • key and value;
  • source of truth;
  • expiration and invalidation;
  • acceptable staleness;
  • miss and stampede behavior;
  • size and eviction policy;
  • behavior when the cache is unavailable.

A cache improves latency and load but adds another consistency state. Never rely on it as the only durable copy.

Partitioning and Replication

Partitioning spreads data or traffic; replication creates additional copies. Choose a partition key that distributes load and supports access patterns. Plan rebalancing and hot-key handling. Define which replica can accept writes and how clients observe replication lag.

Asynchronous Work

Queues smooth bursts and decouple request latency from background work. Bound the backlog, make consumers idempotent, expose queue age, and define retry, dead-letter, and poison-message handling. A growing queue is deferred failure, not successful scaling.

Content Delivery Networks

CDNs move cacheable content closer to users. Define cache keys, freshness, invalidation, privacy, and origin protection. Do not cache personalized responses unless the key and cache controls isolate users correctly.

Reliability Patterns

Timeouts and Deadlines

Every remote call needs a timeout shorter than the caller's remaining deadline. Propagate cancellation where supported. Long timeout chains waste resources after the user has already left.

Retries

Retry only transient failures and operations safe to repeat. Use a small attempt limit, exponential backoff, jitter, and a total deadline. Retry at one layer when possible to avoid multiplicative attempts.

Circuit Breakers and Load Shedding

A circuit breaker temporarily rejects calls when a dependency is failing. Load shedding rejects lower-priority work before saturation collapses all work. Both require observable thresholds and recovery behavior.

Bulkheads and Graceful Degradation

Separate resource pools prevent one workload from exhausting another. Degrade optional features while preserving the core path, and make the degraded state visible.

Redundancy and Recovery

Replication is not backup. Define recovery-point and recovery-time objectives, test restoration, and eliminate single failure domains only where the requirement justifies the cost.

Rate Limiter

Rate limiting protects constrained resources and enforces policy. Decide:

  • identity: user, tenant, credential, IP, route, or combination;
  • limit and time window;
  • burst allowance;
  • whether enforcement is local or shared;
  • behavior when limiter state is unavailable;
  • response headers and client retry guidance.

Common algorithms:

Algorithm Property Trade-off
Fixed window Simple counters Boundary bursts
Sliding log Exact recent history High storage cost
Sliding-window counter Good approximation More state than fixed windows
Token bucket Allows controlled bursts Requires refill state
Leaky bucket Smooth output rate Can delay or reject bursts

Return 429 Too Many Requests when a client exceeds a policy, and use Retry-After when the recovery time is known. Distributed enforcement trades precision, availability, latency, and cost; exact global limits are rarely free.

Real-Time Communication

Choose the least complex mechanism that satisfies the direction and latency needs:

  • polling for infrequent updates;
  • long polling when ordinary HTTP compatibility matters;
  • server-sent events for server-to-client streams;
  • WebSockets for long-lived bidirectional messaging.

Long-lived connections require heartbeats, reconnect rules, resumption or replay, per-client buffers, back-pressure, connection draining, and fleet-wide fan-out.

Observability and Capacity

Measure user outcomes and resource saturation:

  • request rate, error rate, and latency distributions;
  • queue depth and oldest-message age;
  • CPU, memory, storage, connection, and thread-pool saturation;
  • cache hit ratio and eviction rate;
  • database latency, lock contention, and replication lag;
  • dependency success and timeout rates;
  • cost per request, tenant, or workload.

Average latency hides tail behavior. Use percentiles tied to service-level objectives and segment metrics by route, dependency, region, and tenant where useful.

Review Checklist

  • Are requirements and exclusions explicit?
  • Do estimates show the dominant constraints?
  • Are data ownership and invariants clear?
  • Is each added component solving a named problem?
  • Are retries idempotent, bounded, and budgeted?
  • Are queues, pools, and caches bounded?
  • Are overload, partial failure, and recovery covered?
  • Can the design be deployed, migrated, observed, and rolled back?
  • Which assumption is riskiest, and how will it be tested?

See Distributed Systems for consistency, consensus, clocks, and partial failure.