Skip to content

Module 7: Production and reliability

Operate one instance of the task service as though another person depends on it. Package the exact Module 6 artifact, make its behavior observable, define what “reliable” means, and use controlled load and failure experiments to find its real limits.

Learning outcomes

By the end of this module, you should be able to:

  • package and run the service as a constrained container process;
  • separate immutable application content from configuration and persistent data;
  • define liveness, readiness, graceful shutdown, backup, and restore behavior;
  • derive service-level indicators from user-visible outcomes;
  • choose an SLO and calculate its error budget;
  • instrument bounded-cardinality metrics and structured logs;
  • design a repeatable load test and identify saturation from evidence;
  • diagnose an injected failure using telemetry and a runbook.

Preserve the simple deployment model

Deploy:

one containerized application process
    → one persistent SQLite database volume
    → structured logs and metrics

Do not add an orchestrator, service mesh, cache, queue, or multiple application replicas. SQLite and a local volume deliberately make this a single-instance service. Module 8 will evaluate distribution only after this version has a measured limitation.

Build the container from the artifact

The container build consumes the exact wheel and locked runtime dependencies produced in Module 6. Its runtime image should:

  • use a reviewed base image pinned to an immutable digest;
  • contain the installed application and required runtime libraries, not source-control metadata or build tools;
  • run as a dedicated unprivileged user;
  • use an explicit executable entry point so signals reach the service process;
  • write persistent state only beneath one mounted data directory;
  • avoid embedding credentials, local configuration, or the database;
  • include labels for application version and source revision;
  • expose only the application port required by the runtime contract.

Build the image once in CI, record its digest, inspect its contents and configured user, then run smoke tests against that digest. Deployment must promote the same image rather than rebuild its tag.

Keep the root filesystem read-only when the selected runtime supports it. Mount a writable data volume for SQLite and a small temporary filesystem only if the application proves it needs one. Drop unnecessary operating-system capabilities instead of running privileged.

Define the runtime contract

Document and validate configuration at startup:

Setting Purpose Invalid behavior
Listen address and port Network binding Refuse startup
Database path Persistent state location Refuse startup if unsafe or unavailable
Log level and format Operational output Refuse unsupported values
Session lifetime Security behavior Refuse non-positive or excessive values
Request limits and timeouts Resource protection Refuse inconsistent values

Inside a container, the service normally listens on all container interfaces while the runtime decides which host interface is published. Expose it only on host loopback during this local exercise.

Configuration is deploy-time input; the wheel and image remain unchanged between environments. Print effective non-secret configuration at startup, but never credentials or bearer tokens.

Separate startup, migration, and serving

Use explicit commands for:

service migrate
service serve
service check-config

Run migrations as a controlled release step before starting the new process. Do not let every server process race to migrate on startup. The serving command should verify that the schema revision is compatible and refuse readiness when it is not.

Before a migration, take a consistent backup and verify that the restore procedure is known. Application rollback is safe only when the migrated schema remains compatible with the previous application; otherwise recovery requires a forward fix or a tested data restore.

Implement health behavior

Add two endpoints:

Endpoint Question Behavior
/live Is the process able to serve its event loop? Cheap, no dependency checks
/ready Can this instance correctly accept task requests? Check schema compatibility and a bounded database read; report unready after a detected storage failure

Health responses must be fast, unauthenticated, and free of sensitive details. A failing dependency should not make liveness kill an otherwise diagnosable process repeatedly. Readiness may fail until the database is usable. If a mutation detects that storage is unavailable or unwritable, mark the instance unready and require a successful recovery check before accepting normal traffic again.

Test startup with a missing volume, incompatible schema, unwritable database, and invalid configuration. Each should fail clearly without creating partial state.

Handle shutdown as a state transition

When the process receives its normal termination signal:

  1. mark readiness false;
  2. stop accepting new requests;
  3. allow in-flight requests a bounded drain period;
  4. roll back unfinished transactions after the deadline;
  5. close database and telemetry resources;
  6. exit with a meaningful status.

Test this by terminating the container during a deliberately slow request. Verify the client outcome, database state, logs, and total shutdown time. A forced kill should remain exceptional because it cannot run cleanup logic.

Emit useful telemetry

Structured logs

Write one JSON event per line to standard output. Include:

  • timestamp in UTC;
  • severity and event name;
  • request ID;
  • route template, method, status, and duration;
  • authenticated user ID only when operationally necessary;
  • exception type and a sanitized message for unexpected failures.

Do not log raw URLs containing sensitive query values, request bodies, passwords, session tokens, authorization headers, or unbounded stack traces. Keep the security audit trail from Module 5 distinct in purpose even if both use the same transport.

Metrics

Start with a small set:

Metric Useful dimensions
Request count method, route template, status class
Request duration histogram method, route template
In-flight requests none or route template
Database operation duration named operation, outcome
Process CPU and resident memory instance
Database size and volume free space instance
Readiness state instance

Never use task ID, user ID, request ID, raw path, or error message as a metric label. Their unbounded values create a new time series for each request or resource.

This service has one local database hop, so request IDs, logs, and metrics are sufficient initially. Add distributed tracing only when a request crosses multiple independently operated components and correlation becomes a demonstrated problem.

Define user-visible reliability

Choose eligible requests first. For this API, expected contract rejections such as invalid input and missing resources can be valid service responses; server failures, dependency failures, and timeouts are not.

Define two SLIs:

availability = correctly served eligible requests / eligible requests

latency = eligible requests completed within the chosen threshold
          / eligible requests

Define the exact endpoints, response classes, timeout, and measurement source included in each. Averages conceal slow requests; use the proportion below a threshold or inspect latency percentiles alongside it.

Select SLO values from user needs and a measured baseline, not from a universal template. For an SLO target \(T\) over \(N\) eligible requests:

\[\text{error budget} = N \times (1 - T)\]

Write what consumes the budget and what action follows sustained exhaustion. This local project has no external SLA.

Build one operational view

Create a dashboard that answers, in order:

  1. Are users receiving correct responses?
  2. Is latency within the objective?
  3. Is traffic different from the baseline?
  4. Which resource or operation is saturating?
  5. Did a release or configuration change precede the symptom?

Include request rate, error outcomes, latency distribution, in-flight work, CPU, memory, database operation time, database size, free disk, readiness, and deployed image digest.

Alert only on conditions that require timely human action. Prefer sustained user-visible SLO burn or inability to serve over isolated CPU spikes. Every alert needs an owner, urgency, diagnostic links, and a tested runbook.

Establish a repeatable load test

Write a workload model before choosing a tool:

  • operations and their relative frequency;
  • number of distinct users and tasks;
  • request arrival or concurrency model;
  • think time between user actions;
  • payload and database sizes;
  • warm-up, steady measurement, and cool-down phases;
  • success, latency, and resource thresholds.

Run the load generator outside the service container. Use a fresh but representative database snapshot for each comparable run. Record tool version, commands, image digest, configuration, host resources, starting data, and raw results.

Increase load in controlled steps. At each step retain:

  • offered and completed request rate;
  • status and timeout counts;
  • latency distribution;
  • CPU, memory, in-flight requests, database time, and disk behavior.

The capacity boundary is where an objective stops being met, not the highest throughput printed by the tool.

Find one bottleneck

Use this order:

  1. reproduce the limit with the same workload;
  2. identify the resource that saturates as latency or errors rise;
  3. break request time into application and database work;
  4. inspect query plans or profile code only in the implicated layer;
  5. form one hypothesis;
  6. make one change;
  7. rerun the unchanged workload and compare uncertainty, not only best runs.

Reasonable findings include a missing index, an unnecessarily long write transaction, synchronous password hashing consuming worker capacity, or SQLite write contention. Do not add a cache or more replicas unless the measurements identify a problem they actually solve.

Protect and restore the data

Use SQLite's online backup mechanism or another documented consistency-safe method rather than copying a live database file blindly. Define:

  • backup trigger and destination;
  • retention and access permissions;
  • integrity verification;
  • restore steps into an empty data directory;
  • recovery point and recovery time observed in a test.

A backup is unproven until a separate instance restores it, passes migrations, reports ready, and returns expected task and ownership data. Keep the restore test isolated from the active volume.

Run controlled failure exercises

Use disposable data and one failure at a time:

Failure Expected signal and behavior
Normal termination during a request Readiness drops, bounded drain, consistent transaction
Process restart Brief unavailability, persistent tasks and sessions
Database made read-only First mutation fails safely and marks the instance unready; existing data remains intact
Incompatible schema Process refuses readiness with actionable log
Volume approaches configured limit Capacity signal appears before writes fail
Load exceeds capacity Latency or errors consume budget; saturation metric identifies constraint

For each experiment, predict the outcome, inject the fault, detect it from telemetry, recover with the runbook, and compare prediction with observation. Do not inject disk exhaustion or corruption into a shared machine or valuable data volume.

Write one incident record

Treat the most useful failure exercise as an incident. Record:

  • user-visible impact and SLO effect;
  • detection source;
  • UTC timeline;
  • technical cause and contributing conditions;
  • recovery steps and evidence;
  • what worked, what confused diagnosis, and where you were lucky;
  • one or two owned actions that reduce recurrence or detection time.

Focus on system conditions and decisions, not individual blame. An action without an owner and verification condition is only a wish.

Evidence to keep

Your repository should now contain:

  • container build and runtime contract;
  • image version, source revision, digest, and inspection result;
  • liveness, readiness, startup, and shutdown tests;
  • telemetry definitions and a bounded-cardinality check;
  • SLI specification, baseline, SLO, and error-budget policy;
  • dashboard and one actionable alert with runbook;
  • load model, raw results, bottleneck evidence, and before/after comparison;
  • backup and successful restore evidence;
  • failure-exercise notes and incident record.

Completion check

You are ready for the next module when you can:

  • run the exact release image with no source checkout and persistent data outside it;
  • explain why liveness and readiness fail under different conditions;
  • show graceful shutdown leaves data consistent;
  • calculate SLIs and error-budget consumption from retained events;
  • move from a user symptom to the saturated resource using telemetry;
  • reproduce one performance improvement under the same workload;
  • restore the database into a separate working instance;
  • state the single-instance system's measured capacity and first failure mode.

Read only when needed

Continue with Module 8: Distributed systems after the completion check passes.