RAG Is Not a Database Problem

The first architecture diagram is always the same. Documents on the left, an arrow labelled "embed", a vector database in the middle, an arrow to an LLM on the right. The database gets a logo and a name. Everything else is an arrow.

Then retrieval quality is poor, and the team evaluates a different vector database.

Before touching retrieval, build the evaluation set. Fifty to two hundred real questions, each paired with the passages that genuinely answer it.

This is tedious, requires domain knowledge, and produces no demo. It is also the only thing that makes every subsequent decision measurable rather than aesthetic.

Track two things at minimum:

• Recall@k — of the passages that should have been found, how many appeared in the top k?
• Precision@k — of the k passages retrieved, how many were actually relevant?

Recall is the ceiling. If the right passage is not in the retrieved set, no amount of prompting produces a correct grounded answer. Fix recall before anything downstream.

Fixed-size chunking is the default because it is easy, and it is the most damaging default in the pipeline.

Split a document every 512 tokens and you will, reliably, cut a table from its header, separate a definition from the term it defines, and place a procedure's step 4 in a different chunk from steps 1 to 3. The information still exists in your corpus. It is no longer retrievable as an answer.

Split on structure instead:

# Structure first, size second.
sections = split_on_headings(document)      # h1/h2/h3 boundaries

chunks = []
for section in sections:
    if token_count(section) <= MAX_TOKENS:
        chunks.append(section)
    else:
        # Only now fall back to size, and split on paragraphs, never mid-sentence.
        chunks.extend(split_paragraphs(section, MAX_TOKENS, overlap=OVERLAP))

Two additions repay themselves immediately. Prepend the document title and heading path to each chunk, so an orphaned section still carries its context. And keep tables intact even when they exceed the cap, because a truncated table is worse than a long one.

A user asks "why is my deploy stuck?". The document that answers it says "pipeline jobs remain in a pending state when no runner matches the job's tag set."

No shared vocabulary. Embeddings help here — this is exactly what they are for — but they also blur precisely the tokens that matter most in technical corpora: version numbers, error codes, product names, flags.

Neither column wins. Run both and fuse the rankings — reciprocal rank fusion is about fifteen lines of code and needs no tuning:

def rrf(rankings, k=60):
    scores = {}
    for ranking in rankings:              # one list per retriever
        for rank, doc_id in enumerate(ranking, start=1):
            scores[doc_id] = scores.get(doc_id, 0) + 1 / (k + rank)
    return sorted(scores, key=scores.get, reverse=True)

Query rewriting is the other half of this failure. Expanding a terse query into two or three explicit reformulations before retrieval costs one small model call and consistently lifts recall on conversational inputs.

Embedding search is fast and approximate. It compares two vectors computed independently, which means it never actually looks at the query and the document together.

A cross-encoder does. It is far too slow to run across a corpus and entirely practical across fifty candidates.

The pattern is: retrieve 50 by hybrid search, rerank all 50 with a cross-encoder, pass the top 5. In most systems this is the single highest-value change available, and it is roughly an afternoon of work.

Long context windows made it tempting to skip precision entirely. Pass twenty chunks, let the model sort it out.

It does not sort it out. Retrieval quality degrades measurably when relevant content sits in the middle of a long input, and every irrelevant chunk is another opportunity to ground an answer in the wrong place. It is also directly billed.

“Context is not free storage. It is the model's working attention, and it is finite in a way the token limit does not describe.”

• Build an evaluation set of real questions with labelled answer passages
• Fix chunking so complete answers survive as retrievable units
• Add keyword retrieval alongside vector retrieval and fuse the results
• Add a cross-encoder reranker over the shortlist
• Add query rewriting for conversational inputs
• Tune how many chunks reach the prompt, downward
• Only now, consider whether the store itself is a constraint

That last line is where most teams start. By the time you honestly reach it, you will have a benchmark that tells you whether the answer is yes, and in most systems it will not be.