Skip to content

Module 8: Distributed systems

Add one asynchronous boundary only if measurement justifies it. Use a task export that cannot reliably finish within the synchronous request objective, and move that work to a separate worker using the existing database as a durable queue.

The learning exercise may remain on a branch if your real workload does not cross that threshold. Recognizing that distribution is unnecessary is a valid engineering result.

Learning outcomes

By the end of this module, you should be able to:

  • justify an asynchronous boundary from a measured requirement;
  • model a job as explicit durable state transitions;
  • distinguish processing attempts from committed effects;
  • design leases, idempotency, timeouts, retries, and terminal failure;
  • explain the consistency observed by API clients;
  • reason through crashes at every boundary between steps;
  • apply backpressure and measure queue health;
  • identify when the database-backed design reaches its ceiling.

Pass the distribution gate

First implement a synchronous CSV export on an isolated branch and run it against the representative dataset from Module 7. Record:

  • export size and query plan;
  • CPU, memory, database, and elapsed time;
  • effect on normal request latency under concurrent load;
  • request timeout and latency objective;
  • what happens if the client disconnects midway.

Proceed with asynchronous execution only when the export misses a defined request objective, monopolizes serving capacity, or must survive client disconnection. If none applies, keep the synchronous endpoint and document the evidence and revisit condition.

Choose the smallest durable design

When the gate is passed, use:

client → API process → SQLite jobs table
                         ↑       ↓
                   status API  worker process
                                  ↓
                           result stored in SQLite

The API and worker use the same codebase, release image, database volume, domain rules, and deployment host. They are separate processes with independent lifecycles, not separate services.

Why not add a broker yet?

  • the database already provides durable transactions;
  • one worker is enough for the measured workload;
  • queue volume and result size fit the existing capacity;
  • a broker would add another failure, security, backup, and operating boundary.

This design cannot safely scale across hosts while SQLite lives on one local volume. That limit is explicit, not hidden.

Define the asynchronous contract

Add three owner-protected endpoints:

Endpoint Behavior
POST /exports Enqueue one export and return 202 Accepted
GET /exports/{id} Return job status, attempt count, and terminal error code
GET /exports/{id}/result Return CSV only after successful completion

Creation requires the owner's idempotency key. Return a Location header for the status resource and a suggested bounded polling interval. Another user receives 404 for the job and result, matching the task-ownership policy.

Define status responses precisely:

  • queued or running: job is incomplete;
  • succeeded: result endpoint is available;
  • failed: no more automatic attempts will occur;
  • unknown or unauthorized ID: 404.

The enqueue transaction makes the initial queued state visible before 202 is returned. Later polling observes state transitions performed by the worker; acceptance does not imply that a result already exists.

Model durable job state

Create purpose-specific tables rather than a generic workflow engine:

export_jobs(
    id, owner_user_id, request_key, parameters_json, status,
    created_at, available_at, attempt_count,
    lease_token, lease_until,
    started_at, finished_at, last_error_code
)

export_results(
    job_id, content_type, payload, checksum
)

Enforce:

  • one job per (owner_user_id, request_key);
  • export parameters are stored in one canonical representation for idempotency comparison;
  • allowed status values only;
  • non-negative attempt counts;
  • at most one result per job;
  • result ownership through the referenced job;
  • a result exists only for a succeeded job, maintained by the completion transaction.

Use these transitions:

queued ──claim──→ running ──commit result──→ succeeded
                       └──transient failure──→ queued
                       └──permanent/exhausted──→ failed
                       └──lease expires──→ eligible for another claim

Terminal states do not transition automatically. Store stable error codes for clients and sanitized detail for operators; do not persist raw exceptions as API content.

Claim work with a lease

One worker loop is enough initially:

  1. begin a short write transaction;
  2. select one available queued job or an expired running job;
  3. assign a fresh unpredictable lease token and lease deadline;
  4. increment the attempt count and mark it running;
  5. commit before doing expensive work;
  6. process outside the claim transaction.

The claim must be atomic. A conditional update or SQLite write transaction must prevent two workers from both believing they own the same claim.

Use database time for availability and lease comparisons so worker clock differences do not decide ownership. Choose lease duration from measured upper-bound processing time plus margin. If jobs legitimately outlive it, add a lease-renewal protocol; do not begin with heartbeats speculatively.

Every completion or retry update includes both job ID and lease token. A worker whose lease has expired may finish computing, but it can no longer commit an effect.

State the delivery semantics

This design provides at-least-once processing:

  • a worker can crash after claiming;
  • the lease eventually expires;
  • another attempt processes the same logical job.

It does not provide exactly-once execution. Instead, it commits the result once:

  1. compute CSV bytes outside the write transaction;
  2. begin a completion transaction;
  3. verify status, current lease token, and an unexpired lease deadline;
  4. insert the result under the job's unique ID;
  5. mark the job succeeded;
  6. commit both changes atomically.

If lease verification fails, discard the attempt's bytes. The unique result key and transaction prevent two committed results even though computation may occur more than once.

The CSV reflects task state read by the successful processing attempt, not necessarily state at enqueue time. Exact request-time snapshots would require retaining historical task versions or copying input state; do not imply that guarantee without implementing it.

Make enqueueing idempotent

Within one transaction:

  1. authenticate the owner;
  2. normalize the export parameters;
  3. look up (owner_user_id, request_key);
  4. return the original job when key and parameters match;
  5. return 409 when the same key has different parameters;
  6. otherwise insert one queued job and commit;
  7. return 202 only after the row is durable.

Do not publish an in-memory message after the database commit. The durable row itself is the work signal, so there is no database/message dual-write gap.

Define retry behavior

Classify failures rather than retrying every exception:

Failure Action
Temporary database lock or bounded resource pressure Retry after backoff
Worker process loss Lease expiry permits another attempt
Invalid stored job parameters Mark failed permanently
Authorization or ownership invariant failure Mark failed and alert
Result exceeds configured size Mark failed with stable client code
Unknown programming defect Bound retries, retain evidence, then fail

Use exponential backoff with jitter, a maximum attempt count, and an overall job deadline. Derive values from recovery behavior and user expectations; do not retry beyond the point where completion is no longer useful.

A terminal failed row is the dead-letter record for this small system. Provide an operator command to inspect and deliberately requeue a corrected job under a new attempt policy. Do not create another queue just to hold failures.

Be explicit about ordering

Exports are independent. The contract provides no global completion order, even if workers prefer oldest available jobs. Retries and different processing times can reorder completion.

If a future requirement needs per-user ordering, define the ordering key and serialize claims for that key. Global ordering would reduce concurrency and create a system-wide bottleneck; do not add it without a real invariant.

Apply backpressure

A durable queue absorbs a temporary mismatch between arrival and processing, but it does not create capacity. Track:

  • available and running job counts;
  • age of the oldest available job;
  • enqueue and completion rates;
  • processing duration;
  • retry, lease-expiry, and terminal-failure counts;
  • result bytes and database growth.

Define a maximum pending count, oldest acceptable age, and storage budget. When the system cannot accept more work safely, reject new exports with 503 Service Unavailable and a useful Retry-After value while normal task requests remain available.

Start with one worker and one job at a time. Increase concurrency only after measurements show processing capacity is the bottleneck and SQLite contention remains acceptable.

Give API and worker separate reliability objectives

The asynchronous path has two user-visible stages:

acceptance SLI = durably accepted valid export requests / valid export requests

completion SLI = accepted exports completed successfully within deadline
                 / accepted exports

API availability does not prove that the worker is making progress. Dashboard job age and completion separately from request latency. Alert on sustained age or completion-objective burn, not merely on a non-empty queue.

Logs should correlate request ID, job ID, attempt number, and worker ID. Do not log the lease token. Metric labels must use bounded outcomes and error codes, never job or user IDs.

Shut down the worker safely

On normal termination:

  1. stop claiming new jobs;
  2. allow the current attempt a bounded completion period;
  3. commit only while its lease remains valid;
  4. otherwise leave the job running for lease recovery;
  5. close the database and exit.

Do not reset every running job to queued during shutdown: another worker may own it. The lease is the authority.

Test every crash boundary

Use an isolated database and inject process loss at these points:

Crash point Expected recovery
Before enqueue commit No job exists; client can retry key
After enqueue commit, before 202 reaches client Retry returns original job
After claim commit, before work Lease expires; another attempt claims
During CSV computation No result; lease recovery retries
After computation, before completion transaction No result; retry recomputes
During result and success transaction Both commit or neither commits
After completion commit, before worker observes success Job and one result remain succeeded
Stale worker completes after lease reassignment Conditional commit fails; bytes discarded

Also test permanent invalid input, exhausted retries, backpressure, worker shutdown, cross-user access, duplicate idempotency keys, database restart, and result checksum verification.

Measure the cost of distribution

Repeat the Module 7 workload with the API and worker together. Compare:

  • normal request SLOs during exports;
  • export acceptance and completion objectives;
  • SQLite lock time and write latency;
  • CPU, memory, disk, and database growth;
  • operational steps, alerts, and recovery time.

Asynchrony should protect synchronous request behavior and survive disconnection, but it adds delayed results, polling, duplicate attempts, storage, cleanup, and another process to operate. Keep the change only if the measured benefit exceeds that cost.

Record the next ceiling

The database-backed queue should be replaced only when evidence shows a requirement it cannot meet, such as:

  • API and worker must run on different hosts;
  • queue traffic or result storage harms transactional workload;
  • independent scaling or availability is required;
  • routing, fan-out, or retention needs exceed the table design;
  • recovery objectives require a replicated database or external durable store.

At that point, reassess the database, result storage, and queue together. Adding a broker alone does not make the SQLite data volume distributed.

Evidence to keep

Your repository should contain:

  • the synchronous baseline and distribution-gate decision;
  • an architecture decision comparing synchronous, database-queue, and broker options;
  • job schema, transition diagram, and invariant tests;
  • documented delivery, consistency, ordering, timeout, and retry semantics;
  • worker command built from the same release image;
  • crash-boundary and stale-lease tests;
  • queue dashboard, completion SLI, alert, and recovery runbook;
  • before/after workload results and a keep-or-revert decision.

Completion check

The core curriculum is complete when you can:

  • justify whether the asynchronous boundary belongs in the project;
  • explain why processing is at-least-once while the result commits once;
  • demonstrate recovery from every crash boundary without duplicate results;
  • show that stale workers cannot commit after losing a lease;
  • state exactly what clients observe between 202 and completion;
  • explain the absence of global ordering and the cost of adding it;
  • demonstrate backpressure before queue growth threatens the service;
  • name the measurements that would justify a broker or multi-host design.

Read only when needed

Finish with the production service project review and retain the simplest architecture supported by your evidence.