Embeddings, retrieval and RAG · Transformers

Retrieval-augmented generation gives a model the right information at query time: find relevant passages, put them in the prompt, and have the model answer from them. It is the standard way to use private, large or fast-changing knowledge, and in practice a RAG system’s quality is decided mostly by retrieval, not by the generator.

Embedding models

An embedding model maps a text to one vector so that texts with similar meaning land close together, usually measured by cosine similarity:

\cos(u, v) = \frac{u \cdot v}{\lVert u \rVert\, \lVert v \rVert}

Architecturally it is usually a transformer encoder (or an adapted decoder) whose per-token outputs are pooled into one vector - by averaging, or by taking a special token’s output - and normalized.

Training with contrastive learning

Gather pairs that should be close: a question and its answer passage, a title and its article. In a batch of n pairs, each query’s positive is its own passage, and the other n − 1 passages serve as negatives for free (in-batch negatives). The InfoNCE loss is cross-entropy over the batch:

\mathcal{L} = -\log \frac{e^{\,\text{sim}(q,\, d^+)/\tau}}{\sum_{j=1}^{n} e^{\,\text{sim}(q,\, d_j)/\tau}}

It is the same softmax-plus-cross-entropy as in Part 1, with the passages in the batch as the classes. Hard negatives - passages that look relevant but aren’t - teach the finest distinctions and matter a lot. CLIP (Part 14) uses the same loss across images and text.

Bi-encoders vs cross-encoders

	Bi-encoder	Cross-encoder	Late interaction (e.g., ColBERT)
How	encodes query and document separately, compares the vectors	encodes query and document together with full attention between them, outputs a relevance score	keeps one vector per token; matches each query token to its best document token
Speed	fast: document vectors precomputed	slow: one forward pass per (query, document) pair	in between
Accuracy	good	best	very good
Role	first-stage retrieval over millions	reranking the top 20–100	either; needs more storage

The standard design retrieves broadly with a bi-encoder plus keyword search, then reranks a short list with a cross-encoder.

Vector search

Exact nearest-neighbour search compares the query with every vector, O(N·d), which is too slow at scale. Approximate nearest-neighbour (ANN) indexes trade a little recall for large speedups:

HNSW: a multi-layer graph of neighbours; search walks greedily from coarse to fine layers. Fast and accurate, but memory-hungry.
IVF: cluster the vectors, then search only the nearest clusters.
Product quantization (PQ): compress vectors into short codes so more fit in memory.

Libraries and databases (FAISS, pgvector and dedicated vector stores) wrap these with storage, filtering and updates.

Hybrid search

Dense embeddings capture meaning but blur exact strings: part numbers, names, error codes, rare terms. BM25 keyword search catches those. Run both and merge, for example with reciprocal rank fusion, which scores each document by the sum of 1/(k + rank) across result lists. Hybrid search is a strong default.

The RAG pipeline

[embedded content: RAG pipeline · offline indexing, online querying]

Everything the generator sees passes through the retrieval step, which is why recall@k is the first metric to fix.

Indexing (offline):

Parse documents into clean text, keeping structure such as headings and tables.
Chunk them into passages, typically a few hundred tokens, ideally along structural boundaries with some overlap. Attach metadata: source, date, section, permissions.
Embed each chunk; store the vectors and a keyword index.

Querying (online):

Optionally rewrite the question: expand abbreviations, split multi-part questions, resolve references to earlier turns.
Retrieve top-k candidates with hybrid search and metadata filters, including access control.
Rerank and keep the best few.
Build the prompt: instructions, chunks with source ids, the question. Ask for answers grounded in the sources, with citations, and an explicit “not found” when the sources don’t cover it.
Generate, then optionally verify the citations.

Where RAG fails, most common first

Retrieval misses: the answer is in your data but not in the top k - usually chunking, an embedding mismatch or missing keyword search.
Bad chunks: the answer is split across chunks or cut off from its context (“it” with no referent; a table without its header).
Distraction: irrelevant chunks mislead the model, and long contexts dilute attention (“lost in the middle”).
Ignoring the context: the model’s built-in knowledge overrides the retrieved text, or it blends the two.
Injected instructions: retrieved documents can contain text that tries to hijack the model (Part 16).

Evaluating RAG

Measure the two halves separately:

Retrieval: recall@k (is a relevant chunk in the top k?), MRR and nDCG, on a labelled set of questions with known relevant passages.
Generation: faithfulness (is every claim supported by the retrieved text?), answer correctness and citation accuracy, via human review or a carefully validated LLM judge.

Fix retrieval first: if recall@k is low, no generator can recover.

Questions you should now be able to answer

Why are in-batch negatives “free”, and why do hard negatives matter?
Why can’t a cross-encoder be used for first-stage retrieval over 10 million documents?
Give three queries where BM25 beats dense retrieval, and three where dense wins.
Your RAG answers are wrong. How do you decide whether retrieval or generation is at fault?
How would you enforce document permissions in a RAG system?