Monitoring and Observability¶

Monitoring checks known conditions. Observability is the ability to investigate system behavior from emitted evidence. Both should begin with user outcomes and operational decisions, not a vendor stack or a mandate to collect everything.

Start with Service Objectives¶

A service-level indicator (SLI) measures an important outcome, such as successful eligible requests or latency below a threshold. A service-level objective (SLO) sets a target over a window.

availability SLI = successful eligible events / all eligible events
error budget     = 1 - SLO target

Define eligibility, success, measurement point, window, and missing-data behavior. Measure as close to the user as practical. A 99.9% monthly objective allows roughly 0.1% bad eligible events; it is not automatically equivalent to a fixed amount of downtime.

Use error budgets to guide release and reliability decisions, not punish teams. See SRE for broader reliability practice.

Signals and Questions¶

metrics efficiently show rates, distributions, and trends;
logs record discrete events with context;
traces show a request's path and timing across boundaries;
profiles attribute resource consumption to code paths;
events record deployments, configuration changes, failovers, and other state changes.

No fixed number of “pillars” defines observability. Select the signal that answers the question at acceptable cost. The current OpenTelemetry signal documentation tracks its evolving model.

Metrics¶

For request-serving systems, start with request rate, errors, and latency distributions. For queues and batch work, include arrival/completion rate, backlog age, run duration, and last successful completion. For resources, measure utilization and saturation—not only allocation.

Common metric types:

counter: cumulative events, queried as a rate or increase;
gauge: current value that can rise or fall;
histogram: observations distributed into buckets, enabling aggregation and quantiles;
summary: client-calculated distribution statistics with limited aggregation.

Use stable names, base units, and bounded labels. User IDs, request IDs, raw URLs, exception messages, and timestamps create unbounded time-series cardinality. Prometheus instrumentation guidance is a useful primary reference even with another metrics backend.

Logs¶

Emit structured events with timestamp, severity, service, version, operation, outcome, and correlation context where relevant. Log state transitions and unexpected conditions, not every internal step.

never log credentials, tokens, or sensitive payloads by default;
avoid duplicate logging at every layer;
keep messages stable enough to search, with variable data in fields;
attach stack traces to unexpected failures, not normal validation errors;
define retention, access, deletion, and integrity requirements;
sample repetitive low-value events deliberately.

Logs must degrade safely when the sink is slow or unavailable. Unbounded synchronous logging can take down the service it is meant to explain.

Traces¶

A trace contains spans representing operations on a request or workflow. Propagate trace context across synchronous calls and messages, validate it at trust boundaries, and record service, operation, status, duration, and selected safe attributes.

Sampling controls cost. Head sampling decides early; tail sampling can retain traces after observing outcomes but requires buffering and a capable collector. Always preserve enough aggregate metrics to detect problems that unsampled traces miss.

Trace IDs enable correlation; they are not authentication and should not carry user identity by themselves.

Profiles¶

CPU, allocation, heap, lock, and off-CPU profiles answer where resources are consumed. Compare representative intervals and deployed versions. Profiling may add overhead or expose code and data, so control access and validate production settings.

Instrumentation¶

Instrument boundaries where work enters, leaves, queues, retries, or changes durable state. Use consistent semantic conventions and resource identity across services. OpenTelemetry provides vendor-neutral APIs, SDKs, context propagation, and a collector; it does not provide the storage and investigation backend itself.

Instrumentation is production code:

keep overhead bounded;
do not change business behavior when export fails;
test names, units, labels, propagation, and redaction;
version shared conventions;
make telemetry pipeline loss visible.

Alerting¶

An alert should indicate user impact or an imminent condition requiring timely human action. Every page needs:

an owner and destination;
a clear symptom and scope;
links to relevant dashboards and runbooks;
urgency matching the response channel;
enough duration to reject harmless noise;
a way to test routing and resolution.

Page primarily on SLO burn, failed critical workflows, data loss risk, or hard saturation. Use tickets or dashboards for capacity trends and non-urgent hygiene.

Avoid paging separately for every downstream symptom during one incident. Group related alerts and preserve causal context. Review alerts that never fire, always fire, or produce no action.

Dashboards¶

Organize dashboards from user outcome to cause:

SLO and critical workflow health;
traffic, errors, latency, and saturation;
dependencies and queues;
resource and instance detail;
recent deployments and configuration events.

Show distributions and baselines rather than averages alone. A dashboard should support a decision or investigation; delete decorative panels nobody uses.

Telemetry Pipeline Reliability¶

Agents and collectors buffer, process, sample, and export telemetry. Bound their queues and memory, apply back-pressure or shedding, and monitor dropped data, export failures, lag, and cost. Isolate telemetry failure from application correctness.

Plan for backend unavailability and network partitions. Local buffering improves resilience but consumes disk and can replay a burst after recovery.

Privacy, Security, and Cost¶

Telemetry is sensitive operational data and may contain personal information.

minimize collection and redact at the earliest safe point;
enforce tenant and role-based access;
encrypt untrusted transport and protected storage;
audit searches and exports where required;
set retention by diagnostic and legal need;
prevent query and ingestion abuse;
budget cardinality, volume, sampling, and retention.

Cost controls must preserve detection of rare critical failures. Measure cost by signal, service, and owner before applying blanket sampling.

Release and Incident Use¶

Annotate deployments and configuration changes. During progressive delivery, compare candidate and baseline on user outcomes, errors, latency, and saturation; define abort thresholds beforehand.

During an incident:

confirm user impact and scope;
check recent changes and dependencies;
follow metrics from symptom to component;
use traces and logs to narrow the failing path;
profile only when resource evidence points toward code;
mitigate first, then preserve evidence and investigate deeper.

Observability supports diagnosis but does not replace safe rollback, backups, or tested recovery.

Checklist¶

Are SLIs measured at meaningful user boundaries?
Do alerts require a clear and timely action?
Are metrics labeled with bounded dimensions and correct units?
Are logs structured, redacted, and retained intentionally?
Does context propagate across calls and messages safely?
Is telemetry loss and collector saturation observable?
Can dashboards move from symptom to likely cause?
Are releases and configuration changes correlated with behavior?
Are privacy, access, retention, and cost explicitly controlled?
Is unused telemetry removed?

Good observability shortens the path from a user-visible symptom to a safe operational decision.