AI Benchmark Atlas

Score: The Intelligence Index is a weighted mix of 10 evaluations in version 4.3.2. A higher number means better results across those evaluations.

Limits: The evaluation mix changes between index versions, so scores from different versions do not compare directly. Some evaluations in the index are built by Artificial Analysis itself and are not fully public.

Score: Gives each benchmark a health score built from three axes: capability discrimination, anti-saturation, and impact.

Limits: It covers 106 benchmarks taken from the technical reports of 91 models from 2025. Results depend on which scores model makers chose to report.

Limits: The paper tests only 15 benchmarks in math, reasoning, and knowledge with 11 models from four families. Results may not carry over to agent or coding benchmarks.

Limits: Scores depend on task version, prompt format, and few-shot settings. Results from different harness versions or settings may not match.

Limits: The set of scenarios differs between HELM leaderboards and versions. Scores from different HELM releases do not compare directly.

Score: Average score across six benchmarks in the 2024 version: IFEval, BBH, MATH, GPQA, MuSR, and MMLU-Pro.

Limits: The board no longer gets new results. Its public test sets have known contamination risk, and the maintainers said the benchmarks became outdated.

Score: Percent of prompts (or of single instructions) that a rule-based checker marks as followed, at strict or loose matching.

Limits: It has about 500 prompts and 25 instruction types, so it is small. The data is public, so contamination risk is high.

Score: Percent of 2,000 prompts where the output passes every structural check, without constrained decoding.

Limits: The authors designed it to saturate near 100% at the frontier, and top submitted scores are near 99%. Leaderboard entries on Hugging Face are self-reported and not verified.

Score: An Elo-style (Bradley-Terry) rating from human votes. A higher rating means people preferred that model more often.

Limits: Votes reflect user taste and answer style, not only correctness. Published research reports that private testing and data access can favor some providers.

Limits: Scores depend on the dataset mix and prompt settings in each release. Private test sets cannot be checked by outside users.

Score: Percent of 4,326 questions answered correctly. It also reports how often the model answers wrongly instead of declining.

Limits: It tests only short facts, not long answers. The questions were collected against GPT-4, and the public data has contamination risk.

Score: Composite pass^5 percent across tool-use evaluations, with 95% confidence intervals.

Limits: The tasks are private and sold commercially, so outside users cannot reproduce the scores. The task set was refreshed, so older scores may not compare.

Score: Percent of 817 questions answered truthfully (generation) or correctly (multiple choice).

Limits: The data has been public since 2021, so contamination risk is high. The generation score depends on a judge model.

Score: Percent of rubric criteria met on 400 hidden test cases. In the paper, the top model scored 56.1%.

Limits: Only 80 development cases are public, and the 400 test cases are hidden. The public Hugging Face leaderboard has no entries.

Score: A weighted overall score across the 8 environments. A higher score means more tasks completed.

Limits: The environments and model results are from 2023. Newer agent benchmarks test longer and more realistic tasks.

Score: Pass rate on the active task set.

Limits: It is new and few systems have results. Inputs, reference outputs, and environment images are distributed separately from the task cards.

Score: Mean percent of rubric criteria met over 8 runs on 400 hidden tasks.

Limits: A single judge model (Gemini 2.5 Pro) grades all answers, so grader bias is possible. The 400 test tasks are hidden, and only 100 dev tasks are public.

Score: Pass@1: percent of tasks the agent completes on the first try. In the paper, the top agent scored 24.0%.

Limits: Rubric grading uses a judge model. Scores on the Hugging Face leaderboard are self-reported by model makers.

Score: Per-task scores combined into category and overall scores, reported with the cost of each run.

Limits: Many problems come from requests to Ai2's own Asta agents. Scores depend on the provided search tools and the cost budget.

Score: Weighted overall accuracy across test categories. A higher percent means more correct function calls.

Limits: Categories and weights change between versions, so scores across versions do not compare. Some categories need a paid web search API.

Score: Percent of 1,266 questions answered correctly against the reference answer.

Limits: It does not test long answers or unclear user requests. Answers may leak onto the web over time.

Score: Percent of 200 cases completed with every constraint met. No model in the paper scored above 20%.

Limits: It is small, with 200 cases. Prompts are long, more than 4,700 tokens on average.

Score: Weighted percent of check points passed, with core checks weighted more.

Limits: Results are submitted by users, with server-side checks. The README gives two counts (313 tasks in 32 domains and 314 in 33).

Score: Average score for task completion, safety, and handling of injected errors, plus Pass@k and Pass^k over three trials.

Limits: Leaderboard entries on Hugging Face are self-reported from model cards. It has only 38 multi-turn tasks.

Score: Percent of questions answered exactly right. In the paper, people scored 92% and GPT-4 with plugins scored 15%.

Limits: Answers to 166 questions are public, so contamination risk exists for that part. Top scores are self-reported submissions.

Score: Pass rate: percent of tasks where the answer covers at least 75% of the expected claims, scored by a judge model.

Limits: A judge model does the scoring. The public repo has 500 of the 1,000 tasks, and some servers need paid API keys.

Score: Task length, in human-expert time, at which the agent succeeds 50% of the time (80% is also reported).

Limits: Confidence intervals are wide. METR says long estimates are not reliable with its current tasks, and it no longer updates the page actively.

Score: Element accuracy, step success rate, and task success rate against the recorded action sequences.

Limits: It is an offline dataset from 2023, so the saved web pages do not change. A correct path that differs from the human path may score as wrong.

Score: Pass rate: percent of 300 tasks completed. In the paper, GPT-4o scored 47.00%.

Limits: It is small, with 300 tasks. Results are mostly from 2024 models.

Score: Percent of 369 tasks completed. In the paper, people completed 72.36%.

Limits: Some tasks depend on networks and Google Drive. OSWorld-Verified (2025) fixed task errors, and OSWorld 2.0 (2026) adds 108 long tasks.

Score: Average replication score: percent of rubric items met. In the paper, the best agent scored 21.0%.

Limits: It has only 20 papers. A judge model grades the rubrics, and runs are expensive.

Score: F1 score for doing deferred tasks at the right time. In the paper, the best agent scored 65.1%.

Limits: It is text-only and simulated. The paper tests eight models.

Score: Rubric score where 50 means matching the original paper and 70 or more means doing better. In the paper, the best agent scored 21.5.

Limits: It is small, with 40 tasks. A judge model scores the reports, and leaderboard entries are self-reported.

Score: Peak XP (experience points) rate in any 15-second window, or total gold, per task.

Limits: It is a small hobby project with 40 tasks in one game. Results come from the author's own runs.

Score: Skill lift: pass rate with Skills minus pass rate without them. In the paper, Skills raised the average pass rate from 33.9% to 50.5%.

Limits: It has 87 tasks. The Hugging Face leaderboard may be incomplete, and the audited results live in a separate dataset.

Score: Task success rate: percent of tasks with the correct final output.

Limits: The tools, APIs, and data were made by AI and then checked by people. The authors say it is not for ranking models.

Score: Percent of 175 tasks fully completed, plus a partial score from checkpoints.

Limits: The company is simulated and based on software work. It has 175 tasks.

Score: Percent of tasks the agent completes, checked by task-specific scripts.

Limits: Setup needs many real software environments. Most registry results are self-reported by model publishers.

Score: Pass rate and win rate from ToolEval, which uses a model as the judge.

Limits: The live APIs change or fail over time, so results are hard to reproduce. The data was made with ChatGPT.

Score: Average similarity score against milestones and minefields across 1,032 test cases.

Limits: It has 34 tools in one simulated phone-like setting. The user simulator is a model, which can add noise.

Score: Final accuracy plus trajectory scores for tool selection, argument correctness, and order.

Limits: Tools are executable but built for the benchmark, and some trajectories are generated. It uses 1,228 tools.

Score: Percent of 910 tasks completed, checked by task-specific scripts.

Limits: It uses three self-hosted sites, so the web pages are simpler than the open web.

Score: Percent of tasks completed. In the paper, people scored 78.24%.

Limits: It uses self-hosted sites in four domains, so pages are simpler than the open web. The sites need much setup to run.

Score: Overall percent score across 60 tasks, graded by rule checks, environment-state audits and model judges.

Limits: The suite is small, with 60 tasks. Scores depend on the harness (OpenClaw, Claude Code, Codex or Hermes Agent) as well as the model.

Score: Percent of task instances that the agent completes, checked by task validators in the live ServiceNow instance.

Limits: Tasks use only one platform (ServiceNow), so results may not transfer to other enterprise software. Runs need a remote ServiceNow instance, which adds setup work.

Score: Final company outcome (funds) at the end of the simulated year, from a deterministic simulation.

Limits: It is a synthetic simulation, so business rules are simplified. Results are self-reported through pull requests to model repos.

Score: Percent of 225 exercises solved, with a second attempt allowed after the model sees test errors.

Limits: Exercism exercises are public, so contamination risk is high. Top scores reached 88% by late 2025, near the ceiling.

Score: Contest score per problem, turned into a rank and performance rating against human contestants.

Limits: The full set has only 40 problems. Problems come from public contests, so contamination risk exists.

Score: Pass@1: percent of 1,140 tasks where the generated code passes all test cases.

Limits: Tasks are Python only. The data is public, so contamination risk grows over time.

Score: Correctness score from agent graders, shown against median completion tokens; scores only compare within one version.

Limits: The tasks are not public, so outside teams cannot run or check it. Graders are models, and scores change between versions such as 3.1.

Score: Percent of tasks where the agent patch passes the verifier, graded in a clean container.

Limits: The suite is small, with 113 tasks. Published scores use one harness (mini-swe-agent through Pier), so other harnesses may score differently.

Score: Emulator accuracy from test suites and frame-by-frame replay against a reference emulator, including audio.

Limits: It is one task, so it is not a full coding benchmark. The authors say rankings may not show general coding ability.

Score: Pass@k: percent of 164 problems where at least one of k samples passes all tests (pass@1 most often).

Limits: Saturation: current frontier models score near the ceiling. The set is small and public, so contamination risk is high.

Score: Pass@1: percent of problems where the generated code passes all hidden tests in a chosen date window.

Limits: Contamination control needs you to pick problems released after the model cutoff. The newest listed release (v6) ends in April 2025.

Score: Number of the 77 tasks solved with a reward of 0.95 or more (soft solves).

Limits: Few models have results. Dense reward grading gives partial credit, so a solve count and a mean reward can rank models differently.

Score: Percent of test-split problems where the generated code passes all test cases.

Limits: Tasks are simple and public, so top models score high and contamination risk is high. Some original tasks have unclear wording, so a sanitized subset exists.

Score: Percent of problems where the generated CUDA code compiles and passes held-out tests, with separate performance measures.

Limits: It needs NVIDIA GPUs to run. The problem set changes with each release, so compare scores only within one release.

Score: Percent of hidden behavioral tests passed per task, made by agent-driven fuzzing; a task is resolved when all tests pass.

Limits: Tasks are very hard: in the paper no model fully resolved any task. Behavior tests come from fuzzing, so they may miss some behavior.

Score: Percent of 80 main problems (or 338 subproblems) where the generated code passes the test cases.

Limits: The set is small, with 80 main problems. Test data is public, so contamination risk exists.

Score: Percent of 2,294 task instances resolved, meaning the patch passes the fail-to-pass and pass-to-pass tests.

Limits: Tasks come from 12 public Python repositories, so contamination risk is high. Some tasks have unclear issues or flawed tests.

Score: Percent of the 300 issues where the patch passes the unit tests.

Limits: The set is small, about 33 tasks per language on average, so small gaps are noise. Registry results are self-reported.

Score: Pass@1: percent of tasks where the agent patch passes the tests; the public set has 731 original tasks, now 642 after cleanup.

Limits: Only the public subset can be run by outside teams. Scale AI removed 89 public tasks with invalid tests in a later release.

Score: Percent of 500 task instances where the model patch passes the hidden tests.

Limits: Top scores are near the ceiling, and Vals AI stopped running it for that reason. OpenAI also said it no longer reports this benchmark, citing contamination and flawed tests.

Score: Percent of 20 tasks solved; the paper reports frontier agents solve fewer than 30%.

Limits: The suite is small, with 20 tasks. The paper reports reward hacking in 13.8% of rollouts, so verifiers must be strict.

Score: Resolution rate: percent of 89 tasks where the task tests pass, for a given agent harness and model.

Limits: Scores measure the harness and the model together. Newer versions exist (tbench.ai now shows Terminal-Bench 4.0), and 2.0 top scores are above 80%.

Score: Percent of tasks where the task tests pass, for one agent harness and model.

Limits: Scores measure the harness and the model together. Most registry results are self-reported. The tbench.ai site now features Terminal-Bench 4.0.

Score: Percent of trials where the task tests pass, for one agent harness and model. Each task runs several times.

Limits: Scores measure the harness and the model together. It is a continuous benchmark, so version numbers change and scores do not compare across versions.

Score: nDCG@10 (normalized discounted cumulative gain at rank 10) over 1,406 queries.

Limits: The corpus is small, with 8,674 documents. It covers only debate arguments in English.

Score: Average nDCG@10 (normalized discounted cumulative gain at rank 10) across the datasets.

Limits: The datasets are old and public, so contamination risk exists for models trained on them. Most data is English only.

Score: nDCG@10 (normalized discounted cumulative gain at rank 10) across the domains.

Limits: The set is small, with 1,384 queries. Some domains have very few queries.

Score: Truthfulness score: correct answers earn points and hallucinated answers lose points; missing answers score zero.

Limits: Answers to fast-changing questions can go out of date. Mock APIs do not fully match live web search.

Score: ANLS (average normalized Levenshtein similarity) between the predicted answer and the ground truth.

Limits: Top models now score near the ceiling. Test answers are on a hosted server, so full runs need submission.

Limits: It is new. LlamaIndex built it and also sells a document extraction product, so check results from other sources too.

Score: Per-task accuracy or score, averaged across the 12 tasks.

Limits: Several tasks are synthetic retrieval tests that top models now pass. Book tasks use fake entity swaps, which may not match real use.

Score: Accuracy: percent of multiple-choice questions answered correctly (v2); human experts scored 53.7%.

Limits: LongBench v1 contexts are short by current standards. LongBench v2 has only 503 questions.

Score: Overall parsing score per language, computed with the OmniDocBench evaluation pipeline.

Limits: Only the public split is released; the private split needs contact with the authors. The benchmark is new.

Score: Average score across tasks, with each task using its own main metric (for example nDCG@10 for retrieval).

Limits: Results are mostly self-reported, and some models train on the same datasets. The task set grows, so compare scores only on the same benchmark version.

Score: Percent of unit tests passed, overall and per document category.

Limits: The test categories are set by the olmOCR authors, who also build an OCR system. Some test data was made with model help.

Score: Overall score that combines text edit distance, table and formula metrics, and reading-order accuracy.

Limits: Pages are mostly Chinese and English. The page set changed between versions, so compare scores only within one version.

Score: Per-dimension scores from over 169,000 test rules, plus a continuous metric for tables.

Limits: LlamaIndex built the benchmark and also sells a parser (LlamaParse) that it scores. Pages are mostly English public documents.

Score: TRACe metrics: utilization, relevance, adherence and completeness, each compared with labels.

Limits: Labels were made by a GPT-4 annotator, not by people, so label errors are possible.

Limits: The PaddlePaddle team built it and also makes the PaddleOCR parser. It measures robustness to image conditions, not new document types.

Score: Percent of 1,170 queries the agent answers correctly.

Limits: It uses one CRM platform (Salesforce) with synthetic data. A newer version, CRMArena-Pro, extends it.

Score: Pass^k: chance the agent succeeds on a task in all k trials, judged by final database state.

Limits: The original repo tasks are no longer updated; fixed tasks moved to tau2/tau3-bench. Some original tasks had errors.

Score: Pass^k: chance of success in all k trials; best reported pass^1 is about 25.5%.

Limits: It has one domain (banking). Results depend on the retrieval setup the agent uses.

Score: Task completion (pass@1) judged by final database state, with voice quality measured separately.

Limits: Voice agents keep only 30 to 45% of text performance in the paper. Results depend on the simulated user and audio conditions.

Score: Pass^k: chance the agent succeeds in all k trials, judged by final environment state.

Limits: Task fixes changed scores a lot, so compare only within one release. The current release is called tau-bench 2.0, which can confuse names.

Score: Answer score: rubric points earned minus penalties, divided by total possible points; a separate source score checks citations.

Limits: Most of the dataset is internal, with only examples public. Core tasks test single-step work, not agent workflows.

Score: All-pass rate: percent of tasks where every rubric criterion passes; the best initial result was 7.1%.

Limits: Grading uses model judges on rubric criteria. Harvey reports leaderboard scores on a private hold-out set.

Score: Average task score in percent across the 20 tasks, where each task uses its own metric such as accuracy, F1, or ROUGE.

Limits: Covers Chinese law and Chinese text only. Public test data can be in model training data.

Score: Percent of the 100 questions where the system answer contains the correct answer, as judged by GPT-5.2. Groundedness and retrieval accuracy are also reported.

Limits: Small set of 100 questions from one source, the Victorian Criminal Charge Book. The authors (Isaacus) also sell the Kanon 2 Embedder that tops their own results, and an LLM grades the answers.

Score: Average accuracy in percent across the tasks; higher means more correct answers.

Limits: Top models now cluster at about 85 to 89 percent on the Vals AI run, so the score separates frontier models poorly. Public tasks since 2023 can be in training data.

Score: Precision and recall of the retrieved text spans against the expert-marked spans; higher means more exact retrieval.

Limits: Measures retrieval only, not the quality of the final answer. Queries come from LegalBench, so they share its contracts and privacy-policy sources.

Score: Accuracy in percent on multiple-choice questions; open questions get a score from an ensemble of LLM judges against reference answers.

Limits: Mostly Swiss law and two languages. Open-question scores depend on LLM judges, and leaderboard entries on Hugging Face are self-reported.

Score: Micro-F1 and macro-F1 per dataset, averaged across the seven datasets.

Limits: Built for fine-tuned encoder models, not for chat models. English only, and the data is old and public, so contamination risk is high.

Score: Percent of rubric points met, graded by an LLM judge that the authors align with human experts.

Limits: Only part of the data is public (280 of 850 questions). Scores depend on an LLM judge, and the tasks are based on Chinese legal practice.

Score: Percent of weighted attorney rubric criteria the agent passes, as graded by an LLM judge panel.

Limits: Small: 140 tasks from three SaaS (software as a service) contract scenarios. Grades come from LLM judges, and Hugging Face results are not verified.

Score: Accuracy in percent on LegalBench tasks as run by Vals AI.

Limits: Top models cluster between about 85 and 89 percent, so small gaps may not matter. Vals AI chooses the prompts and settings.

Score: Accuracy or macro-F1 in percent on the three-label classification (direct, intermediate, fully evasive).

Limits: Most labels are model-made: the public 16,726-row set uses Eva-4B labels, and only a 1,000-item set has expert labels. Hugging Face results are self-reported.

Score: Accuracy in percent on held-out questions, graded against expert rubrics.

Limits: Version 2 is archived and no longer runs on new models; Version 2.1 replaces it. The test set is private, so only Vals AI can run it.

Score: Percent of questions answered correctly, judged against the gold answer.

Limits: Only 150 of 10,231 questions are public. The authors meant the questions as a minimum standard, so they are fairly direct.

Limits: Each task has its own metric, so there is no single score. Many datasets are older public sets that can be in training data.

Score: Execution accuracy in percent: the share of questions where the computed answer matches the gold answer.

Limits: Data comes from older public earnings reports and is public, so contamination risk is high. Reasoning is mostly short arithmetic programs.

Score: Win rate: the percent of tasks where expert graders prefer the model deliverable to the human expert one, or rate it as equal.

Limits: Only 220 of 1,320 tasks are public. Grading needs human experts or an experimental automated grader, so results are costly to reproduce.

Score: Elo score per work activity or business domain; a higher Elo means the model wins more weighted pairwise benchmark comparisons.

Limits: It reuses scores that others report, so it carries their errors. The snapshot is from June 2026 and needs updates as models change.

Score: Per-task metrics such as accuracy, F1, ROUGE, and MCC (Matthews correlation coefficient), averaged for the ranking.

Limits: Mostly open-weight models run on older public datasets. Task metrics differ, so the average hides large differences.

Score: Pass@1 in percent: the share of tasks where the first agent run passes the hidden tests.

Limits: Small: 87 tasks. New benchmark with results mostly from the authors.

Limits: Each yearly task has its own metrics and test set, so there is no single score. Systems must join the challenge to get official results.

Score: F1 score of predicted codes against the codes in the record, at several levels of detail.

Limits: Needs credentialed access to MIMIC-IV data. Codes in the record are a noisy stand-in for the right clinical decision.

Limits: Different datasets use different metrics. Many of the 11 existing datasets are public and old.

Score: Percent of rubric points earned across conversations; for example o3 scored 60 percent at release.

Limits: A model grader applies the rubrics, so grading noise exists. The data is public, so it can be in training data.

Score: Percent of rubric points earned; the paper also compares models with physician answers on the same tasks.

Limits: The examples were chosen to be hard for OpenAI models, and OpenAI built the benchmark and the top system. About one third are adversarial tests.

Score: Success rate: percent of the 300 tasks the agent completes correctly.

Limits: Small: 300 tasks in 10 categories. The best model reached 69.67 percent at release, so newer models may near the ceiling.

Score: Accuracy in percent: decimal answers count if within 5 percent of the true value; other answers must match exactly.

Limits: Many errors come from the wrong formula or arithmetic, which a real calculator tool avoids. The repo now has a newer 'Verified' version, so check which version a score uses.

Limits: Some benchmarks need gated or private data, so not all results are reproducible. Open-ended tasks use an LLM jury for grading.

Score: Accuracy in percent on the four-option questions.

Limits: Exam-style recall, not real clinical work. The data is public, so contamination risk is high.

Score: Accuracy in percent; most reports use the English USMLE (US Medical Licensing Examination) set.

Limits: Exam-style questions do not show real clinical skill. The data is public since 2020, so contamination risk is high.

Limits: It is a set of separate datasets with their own metrics. The human review needs physicians and is hard to reproduce.

Limits: It gives no benchmark score. Use it to build or test pipelines, not to rank models.

Score: Accuracy in percent on the 1,000 expert-labeled items; a single human scored 78.0 percent.

Limits: Small labeled set of 1,000 items with three answer choices. The data is public since 2019.

Score: AUROC (area under the receiver operating characteristic curve) per task, averaged across tasks.

Limits: Raw audio is not included; users must get each corpus under its own data use agreement. Many tasks have few positive cases.

Score: Pass@1 in percent: the share of tasks the agent drives to the correct end state on the first try.

Limits: The best agent solved 28.0 percent at release. Running it needs a separately gated handbook, and Hugging Face results are not verified.

Score: Percent of the 30 problems answered correctly, often averaged over several runs.

Limits: Only 30 problems, so one problem changes the score by about 3 points. Self-reported scores on Hugging Face reach 97 to 100 percent.

Score: Task score compared with the human best result in the source paper; agents beat it on 4 of 20 tasks at release.

Limits: Small: 20 tasks. Runs need much compute, and results depend on the agent scaffold.

Score: Percent of the 120 semi-private tasks solved within two attempts (pass@2).

Limits: Top systems now reach about 95 percent against a human panel at 100 percent. ARC-AGI-3 now leads the ARC Prize leaderboard.

Score: Agent efficiency compared with humans. 100% means the agent wins every game as efficiently as people do.

Limits: Scores depend strongly on the harness. Provider Adapter and Standard harness results are not comparable.

Score: Average accuracy in percent across the 23 tasks.

Limits: Saturated: the BIG-Bench Extra Hard authors report near-perfect scores on many BBH tasks. Public since 2022.

Score: Average accuracy in percent over 5 runs on the 70 test challenges.

Limits: Small: 71 challenges. Top scores rose from 5.7 percent in the 2025 paper to 83.1 percent on the current v2 leaderboard.

Score: Percent of problems solved, checked automatically against a known answer.

Limits: Only Epoch AI can run the private set, so outside groups cannot reproduce results. OpenAI funded the benchmark and has access to part of the problems.

Score: Accuracy in percent on four-option questions; most reports use the Diamond subset.

Limits: Self-reported top scores now pass 93 percent, above the expert baseline. Some questions have errors, so the true ceiling is below 100 percent.

Score: Accuracy in percent on the 1,319-problem test set.

Limits: Saturated: self-reported top scores reach about 99.6 percent. The data is public since 2021, so contamination risk is high.

Score: Percent of the 33 competition problems where the model's final answer matches the gold answer.

Limits: The set is small, so one problem changes the score by about 3 points. Only final answers are checked, not the proof steps.

Score: Percent of the 2,500 questions the model answers correctly. The authors also report calibration error.

Limits: Questions are public, so contamination risk grows over time. Experts wrote the answers, and some answers may contain errors.

Score: Accuracy per subtask, as the percent of tasks the system answers correctly.

Limits: The original LAB-Bench uses multiple-choice questions, so guessing gives some points. Public task data can leak into training data.

Score: Percent of tasks the model passes, graded against expert rubrics.

Limits: OpenAI built the benchmark and also reports the top model, so independent checks matter. Public access to tasks and rubrics is not confirmed.

Score: Average percent correct across task categories, scored automatically against ground-truth answers without an LLM (large language model) judge.

Limits: Question sets change with each release, so scores from different releases are not directly comparable. Some tasks are harder versions of older public benchmarks.

Score: Percent of problems where the model's final answer matches the reference answer.

Limits: MATH has been public since 2021, so contamination risk is high. Only final answers are checked, not the reasoning.

Score: Percent of competitions where the agent's submission reaches at least a Kaggle bronze medal level against human leaderboards.

Limits: Runs are long and need GPUs, so evaluation is expensive. Agent results vary between seeds, and public Kaggle solutions add contamination risk.

Score: Percent of the 12,032 test questions the model answers correctly.

Limits: Top reported scores are near 88%, so the gap between frontier models is small. The questions are public, so contamination risk exists.

Score: FoldScore, a CASP15-based structure accuracy score; fixed-budget tracks rank by area under the hidden FoldScore learning curve.

Limits: It measures data efficiency under strict rules, not best possible folding accuracy. Hidden-set scoring depends on the organizer.

Score: Percent of the 300 benchmark puzzles the model solves, in single-shot or agentic multi-turn mode.

Limits: Puzzle skill may not carry over to other kinds of reasoning. Agentic runs can take many hours, so they cost a lot to run.

Score: Task success with stepwise verification; the exact headline metric was not confirmed.

Limits: The task set is small. Open-ended research quality is hard to grade objectively.

Score: Normalized effective score averaged over 11 tasks, weighted by progress across a 500k to 50M token budget.

Limits: Each version uses a different task set, so v2 and v3 scores are not comparable. One author built the tasks, and most tasks are hidden.

Score: Two composite scores from 0 to 1: EVA-A for task accuracy and EVA-X for user experience, reported as pass@1 and over multiple trials.

Limits: Simulated users may not match real callers. It covers only three enterprise domains.

Score: Percent of the 6,141 examples answered correctly.

Limits: Most examples come from older public datasets, so contamination risk is high. Answer extraction from free text can add grading noise.

Score: Percent of the 400 questions answered correctly.

Limits: The set is small. Answers depend on live web pages, which can change.

Score: Percent of questions answered correctly under CircularEval.

Limits: An LLM extracts the chosen option from free text, which can add noise. Many questions come from public sources, so contamination risk exists.

Score: Percent of questions answered correctly.

Limits: Questions come from exams and textbooks, so contamination risk exists. MMMU-Pro has only 1,730 questions per setting.

Score: Quality of context-grounded responses to recorded dialogues; the exact headline metric was not confirmed.

Limits: The set has 1,000 dialogues. Grading open responses can be noisy.

Limits: It is new (October 2026) and has a non-commercial license (CC BY-NC-SA 4.0).

Score: Average WER (word error rate), where lower is better, plus RTFx (inverse real-time factor) for speed.

Limits: Test sets are public, so contamination risk exists. WER depends on text normalization rules.

Score: Mask accuracy reported per level; the exact headline metric was not confirmed.

Limits: It covers images only, not video. A single aggregate score hides per-skill differences.

Score: Percent of instructions where the predicted click point falls inside the target element.

Limits: It tests single-step grounding only, not full task completion. Many leaderboard scores are self-reported.

Score: Word error rate on the Hindi test set, where lower is better.

Limits: It covers Hindi only. The dataset is gated, so access needs approval.

Score: Percent of the 2,700 questions answered correctly.

Limits: Video-MME-v2 is a separate, newer benchmark with different scores. Questions are public, so contamination risk exists.

Limits: It is new, and only a few models have registry results. Scores do not compare with the original Video-MME.

Score: Task success rate in simulation.

Limits: It runs in simulation only, so results may not transfer to real robots. The official leaderboard is not yet released.

Limits: Some scores use an LLM judge, which adds grader noise. Different metrics make one overall ranking hard.

Score: Tool selection accuracy, structural consistency, and parameter-filling accuracy of tool calls.

Limits: Spoken queries are synthetic (TTS, text-to-speech), so they may not match real speech.

Score: Scores per subset, such as GPT-judged quality for open QA and accuracy for multiple choice; the leaderboard averages them.

Limits: Much of the audio is synthetic speech. Open-ended subsets use an LLM judge, which adds grader noise.

Score: Average of 22 automatic sub-metrics across five dimensions.

Limits: Automatic metrics use vision models and large multimodal models as judges, which adds grader noise. The set has 289 cases.

Score: Harm score: how much of each harmful task the agent completes, reported with refusal rate.

Limits: Tools are synthetic, so real-world harm is not measured. Public tasks can be trained against.

Limits: It is a suite of several tests with different metrics, so there is no single score. Later versions add tests, so version matters when you compare.

Score: Per mode: recall of known bugs for detect, and percent of vulnerabilities fixed or exploited for patch and exploit.

Limits: Most tasks come from public audits, so contamination risk exists. Detect mode depends on matching reports to known bugs.

Score: Number of the 16 capability flags the agent reaches per bug, each checked by a deterministic oracle.

Limits: The first version covers only 41 V8 bugs. Results on one engine may not apply to other software.

Score: Attack success rate: percent of harmful behaviors where a classifier judges the model complied.

Limits: A trained classifier judges compliance, which adds grader noise. Public behaviors can be trained against.

Score: Harmfulness score of model responses to the 900 harmful test requests, where lower is safer.

Limits: Harmful requests are generated by models, not by real users. An LLM judge rates harm, which adds grader noise.

Score: Percent of tasks with unsafe behavior, and task completion, as judged by an automated LLM judge.

Limits: The set has 150 tasks. The automated judge agrees with humans at about 0.76 to 0.79 F1, so some grader noise remains.

Score: Percent of the 11,435 multiple-choice questions answered correctly.

Limits: Multiple-choice answers test safety knowledge, not safe behavior in open responses. The questions are public, so contamination risk exists.

Limits: It is new and has little outside use. Scores per risk dimension may depend on the judge method.

Score: Percent of vulnerability instances where the agent produces a working PoC or a correct patch.

Limits: The official site now shows SEC-bench Pro, a different 344-instance set. Do not compare Pro scores with original SEC-bench scores.

Score: Harmful response rate: percent of prompts where the model gives policy-violating content, where lower is safer.

Limits: An LLM expanded the prompts, so some may not be realistic. A judge model rates harm, which adds grader noise.