Retrieval-Augmented Generation

Retrieval-augmented generation (RAG) retrieves external evidence and supplies it to a generative model. It improves access to current or private knowledge and enables citations; it does not guarantee that retrieval is relevant or that the model uses evidence faithfully. The foundational formulation is described in the original RAG paper.

Pipeline

sources → parse → chunk/index → retrieve → rerank → assemble context → answer → cite

Begin with keyword search and a small corpus. Add embeddings, hybrid retrieval, reranking, or query rewriting only when evaluation shows a retrieval gap.

Ingestion

Preserve document identity, title, location, version, timestamps, access policy, and source offsets. Remove repeated navigation and broken extraction. Chunk on semantic structure with enough overlap or surrounding context to preserve meaning; there is no universal chunk size.

Index updates and deletions must propagate. A stale chunk should be traceable to its source and reprocessed idempotently.

Retrieval

Sparse search works well for exact terms and identifiers; dense embeddings capture semantic similarity; hybrid search combines them. Metadata filters must enforce tenant and authorization boundaries before evidence reaches the model.

Similarity scores are model- and index-specific, not probabilities. Retrieve a bounded candidate set, rerank when it materially improves relevance, and diversify results when duplicates crowd out evidence.

Answering

Tell the model to use supplied evidence, distinguish evidence from instructions found inside it, cite specific passages, and abstain when evidence is missing or conflicting. Citations must be verified programmatically to refer to retrieved sources; a well-formatted citation can still be unsupported.

Treat documents as untrusted input. They can contain prompt injection, secrets, malicious links, or content unauthorized for the user.

Evaluation

Evaluate stages separately:

• corpus coverage and extraction correctness;
• retrieval recall at k, ranking quality, and authorization;
• answer correctness, completeness, faithfulness, and abstention;
• citation validity and source quality;
• latency, freshness, cost, and failure behavior.

Build cases from real questions, including unanswerable, ambiguous, adversarial, multilingual, and permission-separated examples. LLM judges can scale review but require calibration against humans and should not be the only oracle.

Operations

Version parsers, chunking, embeddings, index configuration, prompts, and models. Monitor ingestion failures, stale sources, empty retrieval, score distributions, latency, abstention, citation use, access violations, and user corrections. Reindexing needs capacity, dual-read validation, and rollback.

Checklist

• Is retrieval necessary versus direct search or long context?
• Can every chunk map to an authoritative source and access policy?
• Are retrieval and generation evaluated independently?
• Can the system abstain and expose conflicting evidence?
• Are citations checked rather than merely generated?
• Are deletion, freshness, reindex, and rollback tested?