Safety and security benchmarks test policy behavior, misuse resistance, harmful capabilities, and cyber tasks. Some datasets are restricted or hazardous. Handle them with care.
How should teams use AI safety and security benchmarks?
Use least-privilege environments, approved datasets, controlled disclosure, and qualified reviewers. Keep capability tests separate from release decisions. A public safety score does not show that your deployed system is safe.
11
Benchmarks in this domain
Sources first · alphabetical collection order
Runnability describes access to a usable repository, dataset, or harness—not whether setup is easy.
AgentHarm
restrictedrunnablehazardous
2026-10-26
Tests
Gives LLM agents explicitly harmful multi-step tasks with tools across 11 harm categories. Tests whether they refuse, and whether jailbroken agents still complete the task.
Tests AI agents on smart-contract security with real audit bugs. Agents must detect, patch, or exploit EVM (Ethereum Virtual Machine) vulnerabilities in sandboxed environments.
Tests LLM agents on real software security tasks in reproducible containers. Tasks are proof-of-concept (PoC) exploit generation and vulnerability patching.
Tests safety alignment on hazardous prompts in chemistry, biology, medicine, pharmacology, physics, and psychology. Prompts derive from real regulations and laws.
Best for
Testing refusal and policy behavior on knowledge-intensive scientific misuse.