Agent benchmarks measure a full system, not only a model. The scaffold, tools, environment, retry rules, and grader all change the result. Use this list to find task environments and harnesses that match your agent.
Match the environment to the work your agent does. Prefer benchmarks that let you inspect the task state, check a fixed end condition, and run the harness yourself. Treat results from different scaffolds as different systems.
33
Benchmarks in this domain
Sources first · alphabetical collection order
Runnability describes access to a usable repository, dataset, or harness—not whether setup is easy.
ACE (AI Consumer Index)
watchpartial
2026-11-03
Tests
ACE (AI Consumer Index) gives models with web search everyday consumer tasks in shopping, food, gaming, and DIY (do it yourself). It checks answers against rubrics and against the web sources the model found.
Best for
Tracking an emerging cross-domain agent benchmark; inspect task coverage before use.
APEX (AI Productivity Index) gives models professional tasks from four jobs: investment banking, consulting, law, and primary care medicine. A judge model grades each answer against an expert rubric.
Tests science research agents on more than 2,400 problems. These cover literature search, code, data analysis, and end-to-end discovery, with fixed search tools.
Tests tool use under many explicit constraints, in 12 categories across resource, behavior, toolset, and response. A program checks each step for constraint breaks.
Best for
Finding constraint violations that task-success-only agent evals miss.
Gives agents 300 checked tasks in service work, multimodal work, and multi-turn dialogue. It grades the full run from traces, audit logs, and environment snapshots.
Best for
A current, source-backed signal for open agent frameworks.
Measures the length of task, in human-expert time, that an AI agent completes with 50% or 80% success. It uses over a hundred software and research tasks.
Best for
Tracking how long a task frontier agents can finish without help.
Tests prospective memory: whether an agent remembers to do a planned task when a future cue occurs. Agents work through a simulated week while doing other work.
Best for
Testing assistants that must reliably follow up when future cues occur.
Gives agents 40 research tasks from 10 science fields, each based on a hidden published paper. A judge model scores the agent's report against an expert rubric.
Best for
Comparing research-agent scaffolds rather than chat-model knowledge alone.
Gives agents tasks from SOPs (standard operating procedures) written by experts in 12 business domains. The agent must follow the steps with the given tools.
Best for
Evaluating process adherence, completion, and tool accuracy in operations workflows.
Gives agents tasks on self-hosted copies of real sites: shopping, forums, code hosting, and content management. It checks whether the task result is correct.
Tests whether an AI agent can finish real, long tasks inside a live CLI (command-line interface) agent harness. Tasks use real tools such as a browser, bash, files, email and calendar, not mock services.
Best for
Comparing general-purpose agent systems on operational work.
Tests whether a web agent can do common knowledge-work tasks in the ServiceNow enterprise platform through the browser. WorkArena++ adds multi-step workflows that need planning and reasoning.
Gives an agent a simulated AI startup for one year, with $200,000 in starting funds, through a CLI (command-line interface). Tests long-horizon choices about tasks, clients, staff, cash flow and deadlines.
Best for
Exploring long-horizon business-agent behavior and trade-offs.