Retrieval Quality

When a system gives you a wrong answer based on searched information, you need to know whether the search failed or the writing failed. Measuring the quality of the search separately tells you exactly that. If the search missed the right documents, no amount of clever writing can fix it. If the search brought back too much irrelevant text, it confuses the writer. By mastering this, you can diagnose and fix your system systematically, rather than guessing. This skill lets you ensure that when a system cites a source, that source actually supports the answer it just gave.

Think of the system as a two-person team: a researcher who finds documents and a writer who drafts the answer. If the researcher does a poor job, the writer has nothing good to work with. But how do you know whose fault the bad answer is? You measure the researcher's work alone, before the writer touches it. You look at the documents the researcher brought and ask two questions: Did they bring the right ones? And did they miss any essential ones? That gives you two numbers: precision and recall. Precision is 'of everything brought, how much was actually relevant?' Recall is 'of everything relevant, how much was actually brought?' If recall is low, the researcher left key documents in the archive. No writer can compensate for missing evidence. If precision is low, the researcher dumped a pile of irrelevant papers on the desk, distracting the writer and crowding out the useful ones.

Imagine you are searching a library for information to write a report on the history of the Indian space programme. You ask a librarian for help. The librarian returns with a stack of books. Precision asks: of those books, how many are actually about the Indian space programme? If half are about NASA, precision is 50%. Recall asks: of all the books in the library that are about the Indian space programme, how many did the librarian bring? If the library has twenty such books and the librarian brought five, recall is 25%. The analogy holds well in retrieval: you always have a fixed set of documents to search, and you retrieve a subset. The total relevant set is usually unknown, but in experiments it is known, so recall is computable. Where the analogy breaks down: in a real library, the librarian can be asked follow-up questions, but in basic retrieval, the search is a single step. Also, relevance is not always black and white here — some documents are more useful than others.

Retrieval quality measures how well a system finds the right documents from a collection, evaluated by checking both the relevance of what was found (precision) and whether anything important was missed (recall).

This concept sits beside retrieval process, which is simply the act of finding documents. It also relates to vector database, which is the specialized storage engine that allows you to find documents based on meaning rather than just matching keywords word-for-word.