Message Queues

Messaging decouples when and where work occurs. It can absorb bursts, distribute work, and publish facts to several consumers. It also introduces delayed results, duplicate delivery, ordering limits, backlog operations, and schema evolution.

Queue, Publish–Subscribe, and Log

• work queue: each message is processed by one consumer from a competing group;
• publish–subscribe: each subscription receives its own copy of a published message;
• durable log/stream: consumers track positions and can replay retained records.

Products often support more than one model. Choose from the workflow and recovery needs, not the product label.

Use synchronous calls when the caller needs an immediate answer and both sides must be available. Use messaging when work may complete later, bursts need buffering, or independent consumers need the same fact.

Message Contract

A useful envelope commonly contains:

{
  "message_id": "01J...",
  "type": "order.shipped",
  "schema_version": 2,
  "occurred_at": "2026-07-25T12:30:00Z",
  "producer": "shipping",
  "correlation_id": "request-123",
  "payload": {"order_id": "o-42"}
}

Keep identifiers stable and payloads bounded. Avoid embedding credentials or unnecessary personal data. Events describe completed facts in past tense; commands request an action and may be rejected.

Delivery Semantics

• at-most-once can lose a message but does not deliberately redeliver;
• at-least-once retries unconfirmed work and can deliver duplicates;
• effectively-once combines at-least-once transport with idempotent business effects;
• exactly-once claims apply only within explicitly documented boundaries.

TCP receipt, broker persistence, consumer delivery, and completed business effect are different milestones. Define which acknowledgement transfers responsibility at each boundary.

Publisher Reliability

A database commit followed by a publish can fail between steps. A transactional outbox writes the business change and an outbox record in one local transaction; a relay later publishes the record.

Publisher confirms establish that the broker accepted responsibility according to its configuration. They do not prove a consumer processed the message. On timeout, the publisher may not know whether acceptance occurred, so republishing needs stable IDs and deduplication.

Consumer Reliability

A robust consumer generally:

1. receives a message;
2. validates envelope and supported schema;
3. checks or establishes idempotency;
4. commits the business effect;
5. acknowledges only after the durable effect succeeds.

Use a unique constraint or atomic state transition at the side-effect store. An in-memory set is lost on restart. If the side effect is an external API, use its idempotency support or reconcile outcomes.

The RabbitMQ reliability guide is a clear primary reference for acknowledgements and publisher confirms, even when using another broker.

Retry and Dead Letters

Classify failures:

• transient: retry with exponential backoff, jitter, and a limit;
• permanent input error: reject or quarantine immediately;
• dependency or systemic outage: slow or pause consumption to prevent a retry storm;
• unknown: preserve evidence and alert an owner.

Do not immediately requeue a failing message in a tight loop. Dead-letter storage is not resolution: record reason, attempt count, timestamps, and source; protect sensitive contents; assign ownership; and provide an idempotent replay path.

Ordering

Global ordering restricts parallelism and is rarely required. Prefer ordering within the smallest key, such as an account or order. Partition by that key and process one ordered stream per partition when the broker supports it.

Retries, redelivery, multiple producers, rebalancing, and failover can still complicate observation. Include entity versions or expected state and make consumers reject, defer, or reconcile stale transitions.

Back-Pressure and Flow Control

A queue is a buffer, not infinite capacity. Bound:

• producer rate or admission;
• message and batch size;
• consumer concurrency and prefetch;
• retry concurrency;
• retention, disk, and memory;
• per-tenant usage.

Queue depth alone is ambiguous. Monitor age of the oldest actionable message, arrival and completion rate, processing latency, retry rate, and resource saturation. If arrival persistently exceeds completion, autoscaling only delays the capacity decision.

Partitioning and Consumer Groups

Partitions permit parallelism and usually define ordering scope. A group assigns each partition to at most one active consumer at a time. More consumers than partitions do not increase parallel consumption for that group.

Rebalancing pauses or moves ownership. Consumers must commit progress and side effects in a safe order and tolerate replay after takeover. Hot keys can dominate one partition even when total throughput looks balanced.

Retention and Replay

Queues often remove acknowledged messages; logs retain records by time or size and let consumers manage positions. Replay is valuable for recovery and new projections but requires:

• compatible historical schemas;
• idempotent consumers;
• controlled replay rate;
• distinction between live and backfill progress;
• sufficient downstream capacity;
• privacy deletion and retention handling.

A retained log is not automatically the authoritative business database or a backup.

Schema Evolution

• prefer additive, optional fields;
• give new fields safe defaults;
• ignore unknown fields where the format permits;
• do not reuse field identifiers with different meanings;
• deploy tolerant consumers before new producers;
• measure old-version usage before removal;
• test real old and new producer/consumer combinations.

A schema registry can enforce structural compatibility but not business meaning.

Choosing a System

Need	Typical fit
Task distribution and rich routing	Queue broker such as RabbitMQ
High-throughput retained event streams and replay	Log platform such as Kafka
Managed elastic queue with minimal operations	Cloud queue service such as SQS
Protocol interoperability	Standards-based broker/client support such as AMQP 1.0

Check the actual product documentation for limits, ordering scope, transaction boundary, retention, availability, and failure behavior. Product names do not guarantee identical semantics. Start with the managed or already-operated system that satisfies the contract.

Security

• authenticate producers, consumers, and administrators separately;
• authorize by topic, queue, tenant, and operation;
• encrypt untrusted network paths;
• rotate short-lived credentials without stopping consumption;
• restrict management interfaces and audit policy changes;
• minimize sensitive payloads and apply retention/deletion rules;
• prevent one tenant from exhausting shared capacity;
• validate untrusted message content before processing.

Messages are an input trust boundary even when produced internally.

Observability and Operations

Measure:

• publish and confirmation errors;
• arrival, delivery, acknowledgement, and completion rates;
• oldest-message age and backlog by partition or tenant;
• processing latency and failures by message type;
• retries, dead letters, duplicates, and poison messages;
• consumer lag, rebalances, and unavailable partitions;
• broker disk, memory, network, replication, and quorum health.

Carry correlation and causation IDs, but avoid treating a message ID as a user identity. Trace asynchronous work across publish, delivery, durable effect, and any emitted messages.

Test broker loss, network ambiguity, duplicate delivery, consumer crash before and after commit, poison messages, full storage, failover, version skew, and replay. Verify business invariants, not only broker availability.

Checklist

• Why is asynchronous messaging better than a direct call here?
• Who owns the message contract and source of truth?
• At what point does each participant acknowledge responsibility?
• Is the business effect idempotent at durable storage?
• What ordering scope is actually required?
• Are retries bounded and dead letters owned?
• Can backlog growth apply back-pressure or admission control?
• Can schemas evolve across independently deployed versions?
• Are replay, repair, privacy, and disaster recovery tested?
• Are end-to-end guarantees stated within their real boundaries?

See Distributed Systems for partial failure, consistency, clocks, and coordination.