5.2 · RAG and retrieval patterns

Don't stuff the whole knowledge base in context. Retrieve the relevant chunks via embeddings or keyword search, and pass only those. Retrieval quality caps answer quality; you can't reason your way past bad retrieval.

Pure vector search captures semantics but misses exact terms (product names, error codes, SKUs). Pure BM25/keyword misses semantic matches. Hybrid combines both, typically with a reranker, and outperforms either alone.

def hybrid_retrieve(query, top_k=20, final_k=5):
    # 1. Two parallel retrievals
    vec_hits = vector_index.search(
        embed(query), top_k=top_k)
    bm25_hits = keyword_index.search(
        query, top_k=top_k)

    # 2. Union (dedup by doc id)
    candidates = {
        d.id: d for d in (vec_hits + bm25_hits)
    }.values()

    # 3. Rerank with a cross-encoder
    #    (scores query+doc together, more accurate than
    #     comparing separate embeddings)
    scored = reranker.score(query,
        [d.text for d in candidates])

    # 4. Return top-N
    ranked = sorted(zip(scored, candidates),
                    key=lambda x: -x[0])
    return [d for _, d in ranked[:final_k]]

Chunks too small lose context; chunks too large dilute retrieval. Overlap between chunks helps preserve boundary-spanning facts. Chunk at semantic boundaries (paragraphs, sections) rather than fixed token counts when possible.

When grounding answers in retrieved documents, instruct the model to cite the specific chunks it used. Gives users verifiability and pressures the model to actually use the retrieved content rather than pattern-match from training.