AI Evaluation Benchmark Atlas

Review 20–50 real outputs after a meaningful change, group failures into a taxonomy, and turn the most frequent high-impact failures into binary checks. Expand toward roughly 100 fresh traces or continue until new traces stop revealing important failure types.

Use public benchmarks to shortlist models, learn task formats, and find runnable harnesses. Use private evals to prove that your system works on your traces, policies, documents, tools, permissions, and business outcomes.

Use deterministic checks first: schema validation, exact match, tool-result assertions, citations, policy thresholds, or execution tests. Use a model judge when the important failure is subjective and recurring, then validate it against human labels.

It is a dated July 2026 census under a public inclusion policy, not a claim that every private, unpublished, or newly released benchmark is captured forever. The catalog currently covers 159 source-linked records; the scored layer is narrower because only benchmarks with inspectable model results can be aggregated.

Raw metrics are never averaged directly. Each result becomes a within-benchmark percentile, benchmark weights reflect participation, discrimination, freshness, and source quality, and the selected use case combines those percentiles. Coverage and confidence remain separate from performance.

There is no context-free best model. Choose the closest work preset, require evidence from multiple benchmarks, exclude irrelevant tests, inspect source provenance, then validate the shortlist on private tasks that match your tools, data, policies, latency, and cost constraints.

Run offline gates before release, sample live traces weekly, and perform new error analysis after model swaps, prompt or tool changes, incidents, complaint spikes, or metric drift.