Site Reliability Engineering¶

Site reliability engineering (SRE) applies software engineering to operating reliable services. It makes reliability a product decision, measures user outcomes, automates repetitive operations, and prepares for failure. It is an implementation of reliability principles, not a required team title.

Reliability as a Product Feature¶

Reliability includes availability, latency, correctness, durability, freshness, and recoverability. The right level depends on user need and cost. Chasing 100% availability produces extreme cost and can prevent safe change while still not guaranteeing perfection.

Define service tiers only when they change concrete expectations such as on-call coverage, recovery objectives, redundancy, and launch requirements.

SLIs, SLOs, and SLAs¶

SLI: a quantitative measure of provided service;
SLO: a target or range for an SLI over a window;
SLA: an agreement that may attach consequences to service performance.

Start from what users care about, then select a measurement point and eligible events. Example:

Over a rolling 28-day window, 99.9% of eligible checkout attempts produce a correct terminal response within 800 ms, measured at the public edge. Client cancellations and rejected invalid requests are excluded.

Document the numerator, denominator, threshold, window, exclusions, data source, and missing-data behavior. Google's SLO guidance provides deeper primary material.

Error Budgets¶

For a success-rate SLO:

allowed bad fraction = 1 - target
budget consumed      = observed bad events / allowed bad events

An error budget makes the reliability/change trade-off explicit. Define policy before the budget is exhausted: progressive rollout requirements, reliability work, change restrictions, and exceptions for urgent security or corrective work.

Use burn rate to detect rapid consumption across short and long windows. Do not wait for the full reporting window to fail, and do not treat a new month as erasing unresolved risk.

Toil¶

Toil is repetitive, manual, automatable, tactical work that scales with service growth and provides little enduring value. Measure it, remove its cause, automate a stable procedure, or deliberately accept it.

Do not automate a process that is unsafe or poorly understood. First document and simplify it, add validation and rollback, then automate. The automation becomes production software with tests, ownership, and monitoring.

On-Call¶

A sustainable rotation needs enough trained people, fair handoffs, protected recovery time, escalation paths, working access, and management support. Pages must be urgent, actionable, and tied to user impact or imminent exhaustion.

Every page should link to:

symptom and affected scope;
relevant dashboards and recent changes;
safe diagnostic steps;
immediate mitigations and rollback;
escalation and communication channels;
conditions for resolution.

Track page volume, after-hours interruption, alert actionability, acknowledgement delay, and recurring causes. Staffing cannot compensate indefinitely for noisy services.

Incident Response¶

Prioritize safety and restoration over perfect diagnosis:

confirm impact and assign severity;
establish an incident lead and communication channel;
stabilize through rollback, isolation, load shedding, failover, or feature reduction;
communicate impact, actions, and next update time;
preserve evidence and test hypotheses;
verify recovery from the user's perspective;
hand off follow-up work with owners.

For larger incidents, separate command, technical operations, and communication roles. Keep a timestamped decision log. Avoid many people making uncoordinated production changes.

Status updates should state known impact, current mitigation, uncertainty, and next update time. Do not speculate about cause externally.

Learning Reviews¶

A useful review reconstructs the conditions that made actions reasonable at the time. Include impact, detection, timeline, technical mechanism, contributing conditions, mitigation, recovery, what helped, and where defenses failed.

Avoid ending at “human error” or a single root cause. Improvements may include removing a hazard, reducing blast radius, earlier detection, safer recovery, clearer ownership, or practice drills. Each action needs an owner, priority, due date, and verification method; fewer strong actions beat a long wish list.

Production Readiness¶

Before launch or major change, review:

ownership, dependency, and architecture boundaries;
SLOs and capacity assumptions;
overload, timeout, retry, and back-pressure behavior;
deployment, compatibility, rollback, and feature controls;
metrics, logs, traces, dashboards, and actionable alerts;
data integrity, backup, restore, and disaster recovery;
identity, secrets, abuse controls, and incident access;
runbooks, escalation, support, and decommissioning.

Use the review to discover risk early, not as a ceremonial launch approval.

Capacity and Overload¶

Forecast demand with uncertainty, identify the first saturated dependency, and preserve operational headroom. Load-test realistic data and traffic shapes, failure states, scale-up delay, and recovery.

Every service needs bounded queues, concurrency, memory, and connection pools. Define load shedding and priority before saturation. Autoscaling adds capacity after a delay and cannot fix hot keys, serial bottlenecks, or exhausted dependencies.

Safe Change¶

Small, reversible changes are a reliability control. Build one immutable artifact, maintain mixed-version compatibility, deploy progressively, observe SLO and saturation signals, and stop automatically when thresholds fail.

Rollback is not sufficient for destructive migrations or irreversible external actions. Use expand–migrate–contract and plan roll-forward recovery.

Resilience and Disaster Recovery¶

Replication, high availability, backup, and disaster recovery solve different problems. Define recovery-point and recovery-time objectives per critical data and workflow. Back up configuration, keys, and external dependencies needed to restore—not just database files.

Run recovery exercises from clean conditions. A backup is only evidence of a write operation until restoration and business correctness are verified.

Controlled Failure Experiments¶

Chaos engineering validates a reliability hypothesis under controlled failure. Start in a safe environment, then increase realism only with observability, abort conditions, bounded blast radius, and informed ownership.

Test likely failures: dependency latency, expired credentials, full storage, zone loss, process pause, message duplication, control-plane unavailability, and recovery of an old leader. Do not inject novelty merely to demonstrate tooling.

Reliability Work Management¶

Balance feature work, reliability risks, toil reduction, and incident actions using SLOs and evidence. Track recurring operational pain and the age of high-risk follow-ups. Reserve engineering capacity for structural improvements instead of relying on heroics.

SRE and product development must share authority and incentives. A separate operations team cannot “own reliability” for software it cannot change.

Checklist¶

Are SLOs based on user outcomes with precise eligibility?
Does error-budget policy trigger agreed decisions?
Are pages urgent, actionable, and sustainable?
Can incidents be coordinated and mitigated safely?
Do learning actions reduce impact or recurrence measurably?
Are capacity, overload, and dependency failures tested?
Can releases stop or recover before broad impact?
Are backups restored against defined objectives?
Is recurring toil being removed rather than normalized?

SRE succeeds when reliability decisions are explicit and ordinary operations do not depend on exceptional people.