RAG (retrieval-augmented generation) evaluation must measure retrieval and answers separately. This list includes embedding and retrieval suites, grounded answer tests, long-context tests, and document parsing benchmarks.
Measure retrieval with labeled query-document pairs and metrics such as Recall@k (share of relevant documents in the top k results). Measure answers separately for faithfulness, relevance, citation support, and correct refusals.
15
Benchmarks in this domain
Sources first · alphabetical collection order
Runnability describes access to a usable repository, dataset, or harness—not whether setup is easy.
ArguAna
specializedrunnable
2026-11-03
Tests
Gives a retriever an argument and asks it to find the best counterargument in a corpus. It is one of the BEIR (Benchmarking-IR) retrieval datasets.
BEIR (Benchmarking-IR) tests zero-shot retrieval across 18 public datasets from different tasks and domains. It checks how well retrievers work on data they did not train on.
Gives a retriever 1,384 real queries from fields such as economics, psychology, math and coding. Finding the relevant documents needs reasoning, not only keyword or topic match.
Best for
Selecting retrievers for difficult research and professional search tasks.
CRAG gives a RAG (retrieval-augmented generation) system 4,409 factual questions in five domains. Mock web search and knowledge-graph APIs supply the retrieval.
Tests long-context models on 12 tasks with average inputs above 100,000 tokens, in English and Chinese. Tasks include book QA, summarization, code debugging and key retrieval.
LongBench tests long-context understanding on 21 datasets in English and Chinese. LongBench v2 uses 503 hard multiple-choice questions with contexts from 8,000 to 2 million words.
MTEB (Massive Text Embedding Benchmark) tests text embedding models on many tasks, such as retrieval, clustering, classification, reranking and semantic similarity. The first release had 58 datasets in 112 languages.
Tests OCR (optical character recognition) systems on 1,403 PDF files with 7,010 unit tests. Tests check text, tables, math, reading order and removal of headers and footers.
Best for
Choosing extraction components before evaluating downstream document RAG.
Tests PDF document parsing on pages from many sources, such as papers, textbooks, newspapers and handwritten notes. It checks text, layout, tables, formulas and reading order.
Tests document parsers on about 2,000 human-verified pages from enterprise documents in insurance, finance and government. It checks tables, charts, content faithfulness, formatting and visual grounding.
Best for
Selecting a parser before measuring retrieval and grounded-answer quality.
Gives 100,000 labeled RAG (retrieval-augmented generation) examples from industry corpora such as user manuals, in five domains. It uses the TRACe metrics to score retrieval and answers.