Monitoring¶

Monitoring is the practice of observing, measuring, and analyzing system behavior to ensure reliability, performance, and availability. In modern software engineering, monitoring has evolved from simple log file inspection and basic alerting to comprehensive observability—the ability to understand a system's internal state by examining its outputs. This discipline is fundamental to operating distributed systems, microservices architectures, and cloud-native applications, where complexity makes traditional debugging methods insufficient.

At its core, monitoring addresses critical questions: Is the system working? How is it performing? Why did it fail? What will break next? Effective monitoring enables teams to detect issues before users do, understand system behavior under load, optimize performance, and maintain service level objectives (SLOs) and agreements (SLAs). It transforms operations from reactive firefighting to proactive management, reducing mean time to detection (MTTD) and mean time to resolution (MTTR).

SLOs, SLIs, and Error Budgets¶

Service Level Indicators (SLIs) are measurable metrics of service behavior (e.g. request success rate, latency p99). Service Level Objectives (SLOs) are targets for those SLIs (e.g. "99.9% availability"). An error budget is the allowed fraction of failures (e.g. 0.1% = 43 minutes of downtime per month). When the error budget is exhausted, teams often freeze feature work and focus on reliability. For SRE practices, incident response, and on-call, see Site Reliability Engineering.

# Example: compute error budget remaining (simplified)
def error_budget_remaining(success_rate: float, target_sli: float) -> float:
    """Target e.g. 0.999 for 99.9%; return fraction of budget remaining."""
    if success_rate >= target_sli:
        return 1.0
    return max(0.0, (target_sli - (1 - success_rate)) / (1 - target_sli))

Modern monitoring encompasses three pillars of observability: metrics (quantitative measurements over time), logs (discrete events with timestamps), and traces (request flows across services). Together, these provide a complete picture of system health, enabling teams to debug complex, distributed systems where traditional debugging tools fall short.

History and Evolution of Monitoring¶

The roots of monitoring trace back to the early days of computing, when operators manually checked system status via console output and log files. The 1990s introduced tools like Nagios (1999), which provided centralized monitoring with alerting capabilities, and Cacti (2001), which focused on time-series data visualization using RRDtool (Round Robin Database Tool).

The 2000s saw the rise of enterprise monitoring solutions: Zabbix (2001) offered comprehensive network and system monitoring, Ganglia (2003) focused on high-performance computing clusters, and Munin (2002) provided resource monitoring with automatic graph generation. These tools primarily monitored infrastructure—CPU, memory, disk, network—using SNMP (Simple Network Management Protocol) and agent-based collection.

The 2010s marked a paradigm shift with the rise of cloud computing, microservices, and containerization. Graphite (2006) and StatsD (2011) popularized metrics collection and aggregation, while InfluxDB (2013) introduced purpose-built time-series databases. Elasticsearch, Logstash, and Kibana (ELK Stack) (2010s) revolutionized log management with searchable, scalable log aggregation.

The modern era (2015-present) is defined by:

Prometheus (2012, CNCF 2016): Pull-based metrics collection with powerful querying (PromQL) and alerting.
OpenTelemetry (2019): Vendor-neutral observability standard unifying metrics, logs, and traces.
Distributed Tracing: Tools like Jaeger (2016) and Zipkin (2012) enable request flow visualization across microservices.
eBPF-based Observability: Tools like Pixie and Cilium Hubble provide zero-instrumentation, kernel-level observability.
AI/ML-Powered Monitoring: Anomaly detection, root cause analysis, and predictive alerting using machine learning.

Today, monitoring has converged with observability—the ability to understand system behavior from external outputs—enabling teams to operate complex, distributed systems with confidence.

Monitoring vs. Observability¶

While often used interchangeably, monitoring and observability represent different paradigms:

Traditional Monitoring¶

Reactive: Alerts when predefined thresholds are breached.
Known Unknowns: Monitors for issues you know to look for (CPU > 80%, error rate > 1%).
Metrics-Focused: Primarily quantitative measurements.
Tool-Centric: Relies on specific tools (Nagios, Zabbix) with limited flexibility.

Observability¶

Proactive: Enables exploration of system behavior to answer novel questions.
Unknown Unknowns: Helps discover issues you didn't anticipate.
Multi-Pillar: Combines metrics, logs, and traces for comprehensive understanding.
Data-Centric: Focuses on collecting rich telemetry data, then querying it flexibly.

The Three Pillars of Observability:

Metrics: Numerical measurements over time (e.g., request rate, latency, error rate).
Logs: Discrete events with context (e.g., application logs, access logs, audit logs).
Traces: Request flows across services, showing the path and timing of operations.

The Fourth Pillar (Emerging):

Profiles: Continuous profiling (CPU, memory, I/O) showing where applications spend time, enabling performance optimization.

Observability is about asking "why" and "what if" questions, while traditional monitoring answers "is it broken?" Modern systems require observability because distributed architectures make it impossible to predict all failure modes.

The Three Pillars of Observability¶

1. Metrics¶

Metrics are numerical measurements collected over time, representing system behavior at a point in time. They are efficient to store, query, and aggregate, making them ideal for dashboards, alerting, and trend analysis.

Types of Metrics:

Counters: Monotonically increasing values (e.g., total requests, errors). Reset on restart.
Gauges: Values that can go up or down (e.g., current memory usage, active connections).
Histograms: Distributions of measurements (e.g., request latency percentiles: p50, p95, p99).
Summaries: Similar to histograms but with quantiles calculated on the client side.

Key Metrics Categories:

Availability: Uptime, error rate, success rate.
Latency: Response time (p50, p95, p99), time to first byte.
Throughput: Requests per second, transactions per second, bytes per second.
Resource Utilization: CPU, memory, disk I/O, network bandwidth.
Business Metrics: Revenue, user signups, conversion rates (often called "golden signals").

Example Metrics:

http_requests_total{method="GET",status="200",endpoint="/api/users"} 15234
http_request_duration_seconds{method="GET",endpoint="/api/users",quantile="0.95"} 0.234
memory_usage_bytes{host="web-01"} 2147483648
cpu_usage_percent{host="web-01"} 45.2

Tools: Prometheus, InfluxDB, Datadog, New Relic, CloudWatch.

2. Logs¶

Logs are discrete events with timestamps, providing detailed context about what happened in the system. They are essential for debugging, auditing, and understanding user behavior.

Log Levels (from most to least severe):

FATAL/CRITICAL: System is unusable, immediate action required.
ERROR: Error events that might allow the application to continue.
WARN: Warning messages for potentially harmful situations.
INFO: Informational messages highlighting progress (default for production).
DEBUG: Detailed information for diagnosing problems (typically disabled in production).
TRACE: Very detailed information, usually only interesting during development.

Structured Logging: Modern best practice uses structured formats (JSON) instead of plain text:

{
  "timestamp": "2025-01-15T10:30:45Z",
  "level": "ERROR",
  "service": "user-service",
  "trace_id": "abc123",
  "message": "Failed to fetch user",
  "user_id": "user-456",
  "error": "database connection timeout"
}

Log Aggregation Challenges:

Volume: High-throughput systems generate terabytes of logs daily.
Storage Costs: Long-term retention is expensive.
Search Performance: Finding relevant logs in large datasets requires indexing.
Correlation: Linking logs across services requires trace IDs or correlation IDs.

Tools: ELK Stack (Elasticsearch, Logstash, Kibana), Loki, Splunk, Datadog Logs, CloudWatch Logs.

3. Traces¶

Distributed tracing tracks requests as they flow through multiple services, showing the complete path and timing of operations. This is critical in microservices architectures where a single user request may traverse dozens of services.

Trace Components:

Trace: The entire request journey (e.g., user clicks "checkout" → payment service → inventory service → shipping service).
Span: A single operation within a trace (e.g., "process payment" in the payment service).
Span Attributes: Key-value pairs providing context (e.g., user_id, order_id, http.method).

Trace Visualization:

Trace: /api/checkout (total: 450ms)
├─ Span: auth-service (50ms)
├─ Span: payment-service (200ms)
│  ├─ Span: charge-card (150ms)
│  └─ Span: update-db (50ms)
├─ Span: inventory-service (100ms)
└─ Span: shipping-service (100ms)

Sampling: To reduce overhead, traces are often sampled (e.g., 1% of requests) in high-throughput systems, with higher sampling for errors.

OpenTelemetry: Industry standard for instrumenting applications, providing vendor-neutral APIs and SDKs for metrics, logs, and traces.

Tools: Jaeger, Zipkin, Tempo, Datadog APM, New Relic, AWS X-Ray.

The Observability Stack: Modern Architecture¶

A typical observability stack in 2025 consists of:

Layer	Component	Popular Tools (2025)
Instrumentation	Application code	OpenTelemetry SDKs, Prometheus client libraries, logging frameworks
Collection	Agents/Exporters	OpenTelemetry Collector, Prometheus exporters, Fluentd, Filebeat
Storage	Time-series DB, Log DB, Trace DB	Prometheus/Mimir/Thanos (metrics), Loki/Elasticsearch (logs), Tempo/Jaeger (traces)
Query & Analysis	Query languages	PromQL (metrics), LogQL (logs), TraceQL (traces)
Visualization	Dashboards	Grafana (unified), Kibana (logs), Jaeger UI (traces)
Alerting	Notification system	Alertmanager, PagerDuty, Opsgenie, Slack
Correlation	Linking data	Trace IDs, correlation IDs, service mesh (Istio, Linkerd)

Data Flow:

Application → OpenTelemetry SDK → OTel Collector →
  ├─ Metrics → Prometheus/Mimir
  ├─ Logs → Loki/Elasticsearch
  └─ Traces → Tempo/Jaeger
         ↓
    Grafana (unified dashboards)
         ↓
    Alertmanager → PagerDuty/Slack

Prometheus¶

Prometheus is the de facto standard for metrics collection and monitoring in cloud-native environments. Developed at SoundCloud in 2012 and donated to the CNCF in 2016, Prometheus has become the foundation of modern observability stacks, with over 1,000 exporters available and integration into Kubernetes, Docker, and most cloud platforms.

Core Mental Model: Pull-Based Metrics Collection¶

Prometheus uses a pull-based model: The Prometheus server scrapes metrics from targets (applications, exporters) at regular intervals (default: 15 seconds). This contrasts with push-based systems (e.g., StatsD) where applications send metrics, offering advantages:

Service Discovery: Automatically discovers targets (e.g., Kubernetes pods).
Reliability: Prometheus controls collection rate, preventing overload.
Simplicity: Applications expose HTTP endpoints; no complex client libraries needed.

Architecture¶

Component	Responsibility	Details
Prometheus Server	Scrapes, stores, and queries metrics	Time-series database with PromQL query language
Exporters	Expose metrics from external systems	Node Exporter (host metrics), cAdvisor (containers), blackbox_exporter (probes)
Pushgateway	Receives metrics from short-lived jobs	Batch jobs, cron jobs that can't be scraped
Service Discovery	Automatically finds targets	Kubernetes, Consul, EC2, Azure, GCP
Alertmanager	Handles alerts from Prometheus	Deduplication, grouping, routing to channels (PagerDuty, Slack)

PromQL: The Query Language¶

PromQL (Prometheus Query Language) is a functional query language for time-series data, enabling powerful analysis and alerting.

Basic Queries:

# Current value
http_requests_total

# Rate over 5 minutes
rate(http_requests_total[5m])

# Error rate percentage
rate(http_requests_total{status="500"}[5m]) / rate(http_requests_total[5m]) * 100

# 95th percentile latency
histogram_quantile(0.95, rate(http_request_duration_seconds_bucket[5m]))

# CPU usage across all instances
100 - (avg(rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100)

Key Functions:

rate(): Per-second average rate over a time range.
increase(): Total increase over a time range.
sum(), avg(), max(), min(): Aggregation functions.
histogram_quantile(): Calculate percentiles from histograms.
label_replace(), label_join(): Modify labels.

Data Model¶

Prometheus stores metrics as time series identified by:

Metric Name: e.g., http_requests_total
Labels: Key-value pairs, e.g., {method="GET", status="200", endpoint="/api/users"}
Timestamp: When the value was recorded
Value: The metric value (float64)

Example Time Series:

http_requests_total{method="GET",status="200",endpoint="/api/users"} @1705315845 → 15234
http_requests_total{method="GET",status="200",endpoint="/api/users"} @1705315860 → 15245
http_requests_total{method="GET",status="500",endpoint="/api/users"} @1705315845 → 12

Storage and Retention¶

Local Storage: Prometheus stores data locally on disk (default: 15 days retention).
Remote Storage: For long-term retention, integrate with Thanos, Cortex, or Mimir.
Compression: Data is compressed over time (older data uses less space).

Alerting Rules¶

Prometheus evaluates alerting rules and sends alerts to Alertmanager:

groups:
  - name: api_alerts
    interval: 30s
    rules:
      - alert: HighErrorRate
        expr: rate(http_requests_total{status="500"}[5m]) > 0.05
        for: 5m
        labels:
          severity: critical
        annotations:
          summary: "High error rate detected"
          description: "Error rate is {{ $value }} errors/sec"

      - alert: HighLatency
        expr: histogram_quantile(0.95, rate(http_request_duration_seconds_bucket[5m])) > 1
        for: 10m
        labels:
          severity: warning
        annotations:
          summary: "95th percentile latency is high"

Exporters Ecosystem¶

Prometheus has a vast ecosystem of exporters for every system:

Node Exporter: Host metrics (CPU, memory, disk, network).
cAdvisor: Container metrics (Docker, Kubernetes).
Blackbox Exporter: Probes (HTTP, TCP, ICMP, DNS).
JMX Exporter: Java application metrics.
PostgreSQL Exporter: Database metrics.
Redis Exporter: Redis metrics.
Custom Exporters: Easy to write in any language (expose /metrics endpoint).

Benefits and Use Cases¶

Strengths:

Industry standard for cloud-native monitoring.
Powerful query language (PromQL) for analysis.
Excellent Kubernetes integration (automatic service discovery).
Large ecosystem (1,000+ exporters).
Open-source and vendor-neutral.

Best For:

Kubernetes and containerized environments.
Microservices architectures.
Teams wanting vendor-neutral, open-source solutions.
High-cardinality metrics (many unique time series).

Limitations:

Pull-based model requires targets to be reachable.
Local storage limits (use Thanos/Mimir for long-term).
High-cardinality metrics can cause performance issues.
Learning curve for PromQL.

Grafana¶

Grafana is the leading open-source platform for metrics visualization and observability dashboards. Originally created by Torkel Ödegaard in 2014, Grafana has become the standard visualization layer for Prometheus, InfluxDB, Elasticsearch, and dozens of other data sources. As of 2025, Grafana Labs (the company behind Grafana) offers both open-source and enterprise solutions, with Grafana Cloud providing managed observability.

Core Features¶

Multi-Data Source Support: Connect to Prometheus, InfluxDB, Elasticsearch, Loki, Tempo, Jaeger, CloudWatch, Azure Monitor, and 100+ data sources.
Rich Visualizations: Graphs, heatmaps, histograms, tables, stat panels, logs, traces, and custom plugins.
Alerting: Built-in alerting engine (since v8.0) with notification channels (Slack, PagerDuty, email).
Templating: Dynamic dashboards with variables (e.g., $datacenter, $service).
Annotations: Mark events (deployments, incidents) on graphs.
Explore Mode: Ad-hoc querying and exploration of metrics, logs, and traces.

Dashboard Example¶

A typical dashboard includes:

Service Overview: Request rate, error rate, latency (p50, p95, p99).
Resource Utilization: CPU, memory, disk, network per instance.
Business Metrics: Revenue, user signups, conversion rates.
Error Analysis: Error breakdown by type, endpoint, user.
Dependencies: Upstream/downstream service health.

Grafana Loki: Log Aggregation¶

Grafana Loki (2018) is a horizontally scalable, log aggregation system inspired by Prometheus. Unlike Elasticsearch, Loki indexes only metadata (labels), storing log content separately, making it cost-effective for high-volume logging.

Key Features:

Label-Based Indexing: Only indexes labels (like Prometheus), not log content.
LogQL: Query language similar to PromQL for logs.
Multi-Tenancy: Isolated tenants for SaaS deployments.
Integration: Seamless integration with Grafana dashboards.

LogQL Example:

# Count errors in the last 5 minutes
sum(count_over_time({job="api"} |= "ERROR" [5m]))

# Top 10 error messages
topk(10, sum by (message) (count_over_time({job="api"} |= "ERROR" [1h])))

Grafana Tempo: Distributed Tracing¶

Grafana Tempo (2020) is a high-scale, cost-effective distributed tracing backend. It stores traces in object storage (S3, GCS, Azure Blob) and integrates with Grafana for unified observability.

Key Features:

Object Storage Backend: Cost-effective (stores traces in S3/GCS).
TraceQL: Query language for traces (similar to PromQL/LogQL).
Integration: Works with OpenTelemetry, Jaeger, Zipkin.
Trace-to-Metrics: Generate metrics from traces (e.g., error rate by service).

Grafana Mimir: Long-Term Metrics Storage¶

Grafana Mimir (2022) is a horizontally scalable, long-term storage system for Prometheus metrics. It replaces Thanos for many organizations, offering better performance and simpler operations.

Key Features:

Horizontal Scaling: Scales to billions of time series.
Prometheus-Compatible: Drop-in replacement, uses PromQL.
Multi-Tenancy: Isolated tenants with resource limits.
High Availability: Replication and redundancy built-in.

Benefits and Use Cases¶

Strengths:

Unified platform for metrics, logs, and traces.
Beautiful, customizable dashboards.
Large plugin ecosystem.
Open-source core with enterprise features available.
Excellent Prometheus integration.

Best For:

Teams using Prometheus (native integration).
Organizations wanting unified observability (metrics + logs + traces).
Multi-cloud environments (supports all major cloud providers).
Teams needing cost-effective log storage (Loki).

ELK Stack (Elasticsearch, Logstash, Kibana)¶

The ELK Stack—Elasticsearch, Logstash, and Kibana—is the most popular open-source log management and analytics platform. Developed by Elastic (founded 2012), the stack has evolved to include Beats (lightweight data shippers) and is now often called the "Elastic Stack."

Components¶

Component	Responsibility	Details
Elasticsearch	Search and analytics engine	Distributed, RESTful search engine built on Apache Lucene
Logstash	Data processing pipeline	Ingests, transforms, and sends data to Elasticsearch
Kibana	Visualization and exploration	Web UI for searching, visualizing, and analyzing data
Beats	Lightweight data shippers	Filebeat (logs), Metricbeat (metrics), Packetbeat (network)

Elasticsearch¶

Elasticsearch is a distributed, RESTful search and analytics engine capable of solving a wide variety of use cases:

Full-Text Search: Fast, relevant search across documents.
Log Analytics: Index and search application logs.
Metrics Storage: Time-series data (though Prometheus is better for pure metrics).
Security Analytics: SIEM (Security Information and Event Management).

Key Concepts:

Index: Collection of documents (like a database).
Document: JSON object stored in an index (like a row).
Shard: Horizontal partition of an index (for scalability).
Replica: Copy of a shard (for high availability).

Data Model:

{
  "timestamp": "2025-01-15T10:30:45Z",
  "level": "ERROR",
  "message": "Database connection failed",
  "service": "user-service",
  "trace_id": "abc123"
}

Logstash¶

Logstash is a server-side data processing pipeline that ingests data from multiple sources, transforms it, and sends it to Elasticsearch.

Pipeline Stages:

Input: Where data comes from (files, Kafka, HTTP, Beats).
Filter: Transform data (parse JSON, add fields, remove fields).
Output: Where data goes (Elasticsearch, S3, Kafka).

Example Configuration:

input {
  beats {
    port => 5044
  }
}

filter {
  if [fields][log_type] == "apache" {
    grok {
      match => { "message" => "%{COMBINEDAPACHELOG}" }
    }
    date {
      match => [ "timestamp", "dd/MMM/yyyy:HH:mm:ss Z" ]
    }
  }
}

output {
  elasticsearch {
    hosts => ["http://elasticsearch:9200"]
    index => "logs-%{+YYYY.MM.dd}"
  }
}

Kibana¶

Kibana provides a web interface for searching, visualizing, and analyzing data stored in Elasticsearch.

Key Features:

Discover: Search and explore data with filters and queries.
Visualize: Create charts, graphs, and maps.
Dashboards: Combine visualizations into dashboards.
Dev Tools: Console for running Elasticsearch queries.
Machine Learning: Anomaly detection and forecasting.

Query Language (KQL):

level:ERROR AND service:user-service AND @timestamp:[now-1h TO now]

Beats¶

Beats are lightweight, single-purpose data shippers:

Filebeat: Ships log files to Elasticsearch/Logstash.
Metricbeat: Collects system and application metrics.
Packetbeat: Network packet analysis.
Heartbeat: Uptime monitoring.
Auditbeat: Audit data collection.

Filebeat Example:

filebeat.inputs:
  - type: log
    paths:
      - /var/log/app/*.log
    fields:
      log_type: application
    multiline.pattern: '^\d{4}-\d{2}-\d{2}'
    multiline.negate: true
    multiline.match: after

output.elasticsearch:
  hosts: ["http://elasticsearch:9200"]
  index: "app-logs-%{+yyyy.MM.dd}"

Benefits and Use Cases¶

Strengths:

Powerful full-text search capabilities.
Flexible data model (JSON documents).
Rich visualization (Kibana).
Large ecosystem (Beats, plugins).
Enterprise features (security, monitoring, ML).

Best For:

Log aggregation and analysis.
Security information and event management (SIEM).
Application performance monitoring (APM).
Full-text search use cases.
Organizations needing enterprise support.

Limitations:

Resource-intensive (requires significant RAM/CPU).
Complex to operate at scale.
Not optimized for pure metrics (use Prometheus).
Licensing changes (some features require paid license).

Distributed Tracing: Jaeger and Zipkin¶

Distributed tracing is essential for understanding request flows in microservices architectures. Two leading open-source tools are Jaeger and Zipkin.

Jaeger¶

Jaeger, developed by Uber and donated to the CNCF, is a distributed tracing platform designed for cloud-native applications.

Architecture:

Jaeger Agent: Receives traces from applications (typically runs as sidecar).
Jaeger Collector: Receives traces from agents, validates, and stores them.
Storage Backend: Elasticsearch, Cassandra, or in-memory (development).
Jaeger Query: API and UI for retrieving and visualizing traces.

Key Features:

OpenTelemetry Integration: Native support for OpenTelemetry.
Service Dependency Graph: Visualizes service relationships.
Trace Comparison: Compare traces to identify regressions.
Sampling: Configurable sampling strategies (e.g., 1% of requests, 100% of errors).

Deployment:

# Kubernetes example
apiVersion: apps/v1
kind: Deployment
metadata:
  name: jaeger
spec:
  template:
    spec:
      containers:
        - name: jaeger-collector
          image: jaegertracing/jaeger-collector:latest

        - name: jaeger-query
          image: jaegertracing/jaeger-query:latest

Zipkin¶

Zipkin, developed by Twitter and open-sourced, is a distributed tracing system focused on latency problem troubleshooting.

Architecture:

Instrumentation Libraries: Client libraries for various languages.
Zipkin Collector: Receives and validates traces.
Storage: In-memory, MySQL, Elasticsearch, Cassandra.
Zipkin UI: Web interface for trace visualization.

Key Features:

Simple Architecture: Easier to deploy than Jaeger.
Broad Language Support: Libraries for Java, Python, Go, Node.js, etc.
Service Map: Visualizes service dependencies.
Trace Search: Search traces by service, operation, duration.

Comparison: Jaeger vs. Zipkin¶

Feature	Jaeger	Zipkin
Architecture	More complex (agent, collector, query)	Simpler (collector, storage, UI)
Storage	Elasticsearch, Cassandra, in-memory	In-memory, MySQL, Elasticsearch, Cassandra
OpenTelemetry	Native support	Via adapters
UI	More modern, feature-rich	Simpler, focused
Sampling	Advanced strategies	Basic sampling
Best For	Cloud-native, Kubernetes	Simpler deployments, legacy systems

OpenTelemetry: The Observability Standard¶

OpenTelemetry (OTel) is a vendor-neutral, open-source observability framework that provides APIs, SDKs, and tools for instrumenting, generating, collecting, and exporting telemetry data (metrics, logs, and traces). Launched in 2019 by merging OpenTracing and OpenCensus, OpenTelemetry has become the industry standard for observability instrumentation.

Core Components¶

Component	Responsibility	Details
APIs	Language-specific APIs	Define how to instrument applications
SDKs	Language implementations	Provide the actual instrumentation code
Instrumentation Libraries	Auto-instrumentation	Automatically instrument common frameworks (HTTP, gRPC, databases)
Collector	Telemetry processing	Receives, processes, and exports telemetry data
Exporters	Backend integration	Send data to Prometheus, Jaeger, Datadog, etc.

Architecture¶

Application Code
    ↓ (instrumentation)
OpenTelemetry SDK
    ↓ (export)
OpenTelemetry Collector
    ↓ (exporters)
Prometheus / Jaeger / Datadog / etc.

Instrumentation Example (Go)¶

package main

import (
    "go.opentelemetry.io/otel"
    "go.opentelemetry.io/otel/exporters/jaeger"
    "go.opentelemetry.io/otel/sdk/trace"
)

func main() {
    // Create Jaeger exporter
    exporter, _ := jaeger.New(jaeger.WithCollectorEndpoint(
        jaeger.WithEndpoint("http://jaeger:14268/api/traces"),
    ))

    // Create tracer provider
    tp := trace.NewTracerProvider(
        trace.WithBatcher(exporter),
        trace.WithResource(resource.NewWithAttributes(
            semconv.SchemaURL,
            semconv.ServiceNameKey.String("my-service"),
        )),
    )
    defer tp.Shutdown()

    otel.SetTracerProvider(tp)

    // Use tracer
    tracer := otel.Tracer("my-service")
    ctx, span := tracer.Start(context.Background(), "operation")
    defer span.End()

    // Your application code
}

Benefits¶

Vendor Neutral: Instrument once, export to any backend.
Standardization: Industry-wide standard reduces lock-in.
Auto-Instrumentation: Automatic instrumentation for common frameworks.
Rich Context: Propagates trace context across services.
Active Development: Rapidly evolving with broad industry support.

Monitoring Best Practices¶

Effective monitoring requires following established best practices:

1. Define SLOs and SLIs¶

SLO (Service Level Objective): Target level of service (e.g., 99.9% uptime).
SLI (Service Level Indicator): Measured metric (e.g., error rate, latency).
SLA (Service Level Agreement): Contract with users (e.g., 99.9% uptime or refund).

Example: "99.9% of requests should complete in <200ms" (SLO). Measure p99 latency (SLI).

2. Use the Four Golden Signals¶

From Google's Site Reliability Engineering (SRE):

Latency: Time to serve a request (p50, p95, p99).
Traffic: Demand (requests per second, concurrent users).
Errors: Error rate (4xx, 5xx responses, exceptions).
Saturation: Resource utilization (CPU, memory, disk, network).

3. Implement Proper Alerting¶

Alert on Symptoms, Not Causes: Alert on "high error rate" not "database connection pool exhausted."
Avoid Alert Fatigue: Use alerting hierarchies (critical → warning → info).
Page on What Matters: Only page on user-impacting issues.
Use Runbooks: Document how to respond to alerts.

Alerting Rules:

Critical: User-facing issues (high error rate, downtime).
Warning: Degraded performance (high latency, resource saturation).
Info: Non-urgent issues (deprecated API usage, capacity planning).

4. Instrument Everything¶

Application Metrics: Business logic, request rates, errors.
Infrastructure Metrics: CPU, memory, disk, network.
Dependencies: Database, cache, external APIs.
Business Metrics: Revenue, user signups, conversion rates.

5. Use Structured Logging¶

JSON Format: Machine-readable, easy to parse.
Consistent Fields: timestamp, level, service, trace_id, message.
Context: Include relevant context (user_id, request_id, etc.).
Avoid PII: Don't log sensitive data (passwords, credit cards).

6. Implement Distributed Tracing¶

Trace All Requests: Instrument all services in the request path.
Use Trace IDs: Propagate trace IDs across services.
Sample Appropriately: Sample 1% of requests, 100% of errors.
Correlate with Logs: Include trace IDs in logs for correlation.

7. Monitor Dependencies¶

Health Checks: Monitor upstream services (databases, APIs).
Circuit Breakers: Fail fast when dependencies are down.
Timeout Configuration: Set appropriate timeouts.
Retry Logic: Implement exponential backoff.

8. Use Dashboards Effectively¶

Service Dashboards: One dashboard per service (overview, errors, latency, dependencies).
Team Dashboards: High-level metrics for team visibility.
Executive Dashboards: Business metrics (revenue, users, growth).
On-Call Dashboards: Critical metrics for incident response.

9. Implement Log Retention Policies¶

Hot Storage: Recent logs (last 7 days) in fast storage (Elasticsearch).
Warm Storage: Older logs (7-30 days) in cheaper storage (S3).
Cold Storage: Archive logs (>30 days) for compliance.
Cost Optimization: Use log sampling, compression, and tiered storage.

10. Continuous Improvement¶

Review Alerts: Regularly review and tune alerting rules.
Post-Mortems: Learn from incidents, update monitoring.
Capacity Planning: Monitor trends, plan for growth.
Tool Evaluation: Regularly evaluate new tools and practices.

Modern Trends: Monitoring in 2025¶

The monitoring landscape continues evolving:

eBPF-Based Observability¶

eBPF (extended Berkeley Packet Filter) enables kernel-level observability without application instrumentation:

Pixie: Auto-instrumentation using eBPF (no code changes).
Cilium Hubble: Network and security observability for Kubernetes.
Falco: Runtime security monitoring using eBPF.

AI/ML-Powered Monitoring¶

Anomaly Detection: Machine learning identifies unusual patterns.
Root Cause Analysis: AI suggests likely causes of incidents.
Predictive Alerting: Predict issues before they occur.
Intelligent Sampling: Adaptive sampling based on error rates.

Observability as Code¶

Infrastructure as Code: Define dashboards, alerts, and SLOs in code (Terraform, Pulumi).
GitOps for Observability: Manage monitoring configuration in Git.
Version Control: Track changes to dashboards and alerts.

Serverless and Edge Monitoring¶

Lambda Monitoring: CloudWatch, Datadog Serverless, New Relic.
Edge Observability: Monitor CDN, edge functions, IoT devices.
Distributed Tracing: Trace requests across serverless functions.

Cost Optimization¶

Log Sampling: Sample logs to reduce storage costs.
Metrics Aggregation: Aggregate high-cardinality metrics.
Tiered Storage: Hot/warm/cold storage for logs and traces.
Right-Sizing: Use appropriate retention periods.

Integration with DevOps and SRE¶

Monitoring is integral to DevOps and Site Reliability Engineering (SRE):

CI/CD Integration: Monitor deployments, track success rates, detect regressions.
Canary Deployments: Monitor canary instances, compare metrics, rollback if issues.
Chaos Engineering: Intentionally break systems, verify monitoring detects issues.
Incident Response: Use monitoring for on-call, incident triage, and post-mortems.
Capacity Planning: Use metrics to plan for growth, right-size infrastructure.

Example SRE Practices:

Error Budgets: Allow a certain percentage of errors (e.g., 0.1% for 99.9% SLO).
Blameless Post-Mortems: Learn from incidents without blame.
Toil Reduction: Automate repetitive tasks, focus on engineering work.

Conclusion¶

Monitoring has evolved from simple log inspection to comprehensive observability, enabling teams to operate complex, distributed systems with confidence. The three pillars of observability—metrics, logs, and traces—provide complementary views of system behavior, while modern tools like Prometheus, Grafana, and OpenTelemetry standardize instrumentation and data collection.

As systems become more distributed and cloud-native, observability becomes not just a nice-to-have but a necessity. Effective monitoring enables proactive issue detection, faster incident resolution, and data-driven decision-making. By following best practices—defining SLOs, implementing proper alerting, using structured logging, and instrumenting comprehensively—teams can build reliable, performant systems that meet user expectations.

The future of monitoring lies in AI-powered anomaly detection, eBPF-based zero-instrumentation observability, and observability as code, making it easier than ever to understand and operate complex systems at scale.