These benchmarks cover code generation, GitHub issue repair, terminal work, debugging, and multi-hour software tasks. The most useful ones run the generated code against tests in a fixed environment.
Use repository repair sets for end-to-end coding agents. Use terminal benchmarks for agents that work in a shell. Use new problem sets for code generation. Record the scaffold, tool access, time budget, and pass rule with every result.
21
Benchmarks in this domain
Sources first · alphabetical collection order
Runnability describes access to a usable repository, dataset, or harness—not whether setup is easy.
Aider Polyglot
recommendedpartial
2026-10-26
Tests
Gives a model 225 hard Exercism coding exercises in six languages (C++, Go, Java, JavaScript, Python, Rust). Tests whether the model can edit code so the unit tests pass.
Gives an AI system hard optimization problems from AtCoder Heuristic Contests, such as routing and scheduling. Tests long, iterative algorithm engineering with no known exact solution.
Gives a model 1,140 Python tasks that need calls to functions from 139 libraries in 7 domains. Tests whether the model follows complex instructions and uses many tools in code.
Cursor's internal set of coding-agent tasks, made from real Cursor sessions of its engineers. Tests correctness, code quality, efficiency and interaction behavior.
Gives a coding agent 113 original long-horizon tasks from active open-source repositories in TypeScript, Go, Python, JavaScript and Rust. Each task runs in an isolated environment with program verifiers.
Gives a model contest problems from LeetCode, AtCoder and Codeforces, sorted by release date. Tests code generation, self-repair, code execution and test output prediction.
MBPP (Mostly Basic Programming Problems) gives a model 974 short Python tasks for entry-level programmers. Each task has a text description and three test cases.
Gives a model handcrafted CUDA programming problems for kernels, runtime APIs, GPU libraries and DSL (domain-specific language) kernel authoring. Tests correctness and runtime performance of the generated code.
Best for
Evaluating coding systems that optimize kernels or write CUDA.
Gives an agent only a compiled program and its documentation. Tests whether the agent can build a full codebase that behaves like the original program.
Gives a model research coding problems written by scientists in 16 natural-science subfields. Each main problem splits into subproblems that need knowledge recall, reasoning and code.
Gives a model a real GitHub issue and the Python repository at that commit. Tests whether the model can write a patch that makes the hidden tests pass.
Gives a coding agent long software engineering tasks from 41 maintained repositories. Held-out and commercial subsets are private to reduce contamination.
Best for
Frontier SWE agents and benchmark saturation checks
Gives an agent 20 very long software engineering tasks that take hours and millions of tokens. Each task has its own environment, a human reference solution and layered verification.
Gives an agent 89 hard tasks in containerized terminal environments, such as debugging code and fixing security issues. Tests whether the agent can finish real work through the shell.