Reranking

Reranking is the single highest-return addition to a basic RAG pipeline. It turns a cheap, slightly sloppy first search into a precise final list — the difference between the generator pulling in irrelevant chunks and getting exactly the evidence it needs. Master this and you unlock: Context Precision and Recall (the metrics that prove your retrieval is good), then Multi-Document Retrieval (weighing several sources), then Admissible Evidence Spans (the regulated-world requirement that answers trace to specific verified passages). If you want to build a production-grade RAG system, reranking is the step that separates a demo from something you'd ship.

When you search a large collection of documents for a query, the first pass is deliberately fast but rough. It might bring back a hundred candidate chunks — many are on-topic, some are close but not quite right, others are clear misses. Reranking is the second, slower pass: it takes that pile of candidates, re-reads each one against your exact query, and produces a refined, ordered list — maybe just the top five — that you actually hand to the language model. The key idea is that you don't ask the cheap first pass to be perfect; you ask it to be fast and recall as many plausible candidates as it can, then you let a more careful, more expensive model decide which of those candidates are truly most relevant. This is the 'cheap wide retrieval, expensive narrow rerank' pattern. Think of it as a two-stage filter: the first stage is a rough sieve that catches everything that might be relevant, and the second stage is a fine mesh that keeps only the best.

Imagine you're a hiring manager at a large company with thousands of résumés for a single software developer position. Your first pass is a quick keyword scan: does the résumé mention 'Python' or 'API design'? That scan is fast and covers every single résumé, but it's inaccurate — it might flag a résumé because the word 'Python' is mentioned in a hobby, or miss a brilliant candidate who didn't use that exact keyword. You now have a shortlist of fifty résumés. The second pass is a manual, careful read of each one, comparing it directly against the job description. You check not just 'mentions Python' but 'has built a Python web service in production' and 'has designed REST APIs'. This careful read is slow — you can't do it for thousands — but it's far more accurate. After this, you have your final five candidates to interview. Reranking works exactly this way. The first retrieval is the keyword scan: fast, covers the whole corpus, but rough. The reranker is the careful human reader: it looks at each chunk *together with* the query and decides, with high precision, which ones are truly most relevant. Where the analogy breaks down: the reranker isn't human, it's another machine-learning model, so it doesn't have human judgment — it has been trained on millions of query-document pairs to *predict* what a human would find relevant. And 'interviewing' the final chunks is the generator's job, not the reranker's.

Reranking is the process of taking an initial set of candidate documents (or chunks) retrieved by a fast but rough method, and re-scoring them with a more expensive, more accurate model that considers the query and document together, in order to select and order the most relevant subset before passing it to the generator.

You don't know embeddings or vector search yet, so think of the first retrieval stage as a black box that returns a list of candidates. Your library notes say that reranking is a key part of the RAG pipeline: ingest, chunk, embed, retrieve, rerank, generate. This concept pairs naturally with Context Precision and Recall — reranking improves precision by removing irrelevant chunks that would otherwise eat the model's context budget and distract it. It also supports Admissible Evidence Spans: when answers must cite specific verified passages, the reranker ensures the generator only sees passages that are truly on-topic, making citation and compliance much easier to enforce. It is distinct from Agentic Retrieval, where the model decides to search again in a loop — that is a different layer. Reranking is a one-shot refinement of the initial set.