The pipeline has independent failure points
A typical RAG system ingests documents, segments them, creates searchable representations, retrieves candidates, reranks them, builds a context window, and asks a model to answer. Evaluate each stage separately. If the needed passage never enters the candidate set, prompt engineering cannot recover it. If the passage is retrieved but the model ignores it, improving the index may not help.
Mental model: retrieval determines the evidence ceiling; generation determines how faithfully the system uses that evidence.
Chunking creates the searchable unit
Chunks that are too small lose definitions, qualifiers, and table context. Chunks that are too large dilute the matching signal and consume the context window. Structural boundaries—headings, paragraphs, code blocks, records, or conversation turns—often matter more than a fixed token count. Useful metadata preserves source, date, permissions, section, and relationships needed later for filtering or citation.
Dense, sparse, and hybrid retrieval
Sparse retrieval is strong when exact terms, identifiers, and rare names matter. Dense retrieval can connect paraphrases and semantic similarity. Neither dominates every query. Hybrid systems combine their candidate sets or scores, then rerank with a more expensive model. The right design depends on query type and corpus, so evaluation should segment results rather than report only one average.
Graphs help with relationships, not by default
Graph retrieval is valuable when answers depend on multi-hop relationships, global structure, provenance, or entities mentioned across documents. It introduces extraction error, entity resolution, schema choices, graph maintenance, and more complex retrieval. A graph is not automatically better than well-designed metadata and hybrid search.
Ask whether the evaluation contains questions that truly require relationships. If most answers live in one passage, a graph pipeline may add cost without increasing useful recall.
Grounding is stricter than citation formatting
A system is grounded when its claims are supported by supplied evidence. Adding citation markers after generation does not guarantee support. Evaluate whether each material claim is entailed by the cited passage, whether the source is authoritative and current, and whether contradictory evidence was omitted. The system should be allowed to abstain when evidence is insufficient.
Measure retrieval before answer quality
Create a test set with questions, relevant evidence, acceptable answers, and unanswerable cases. Retrieval metrics can measure whether at least one relevant passage appears in the top results and where it ranks. Generation evaluation should separately measure factual support, completeness, citation correctness, and refusal quality. Human review remains important where wording or authority matters.
Latency and cost must include embedding, filters, query rewriting, multiple retrieval calls, reranking, and model generation. Freshness tests should confirm that updates and deletions propagate, especially when documents have access controls.
Questions to ask when reading a paper
- Where is the gain? Candidate recall, reranking, context construction, generation, or the dataset itself?
- What corpus is used? Size, domain, duplication, update rate, and access restrictions?
- Are baselines tuned fairly? Equal context budget, candidate count, model, and latency?
- Do questions require the proposed structure? Especially for multi-hop or graph methods.
- How are unsupported answers handled? Abstention, conflicting evidence, and citation verification?
A disciplined starter system
Begin with a small trusted corpus and a labeled set of real questions. Preserve document structure and source metadata. Establish a sparse baseline, then add dense retrieval and reranking one component at a time. Inspect misses manually. Track answer support at the claim level and include unanswerable questions. Only introduce query agents or graphs when the error analysis shows which failure they solve.
Bottom line
Good RAG is not “put documents in a vector database.” It is evidence engineering: defining searchable units, retrieving with appropriate signals, preserving provenance, and measuring whether the final answer stays inside the evidence.