Healthcare benchmarks cover medical knowledge, clinical reasoning, biomedical research, safety, and agent workflows. A high score does not show that a system is ready for clinical use.
How should healthcare AI benchmarks be interpreted?
Measure factual knowledge and clinical workflow reliability separately. Validate with qualified clinicians, patient safety limits, current guidance, abstention behavior, and the exact decisions the system may affect.
15
Benchmarks in this domain
Sources first · alphabetical collection order
Runnability describes access to a usable repository, dataset, or harness—not whether setup is easy.
BioASQ
recommendedpartial
2026-10-26
Tests
A yearly challenge that tests biomedical semantic indexing and question answering. Systems label PubMed articles with MeSH (Medical Subject Headings) terms and answer expert questions.
Tests LLMs (large language models) on clinical decisions from MIMIC-IV hospital records: diagnoses, procedures, lab-test orders, and prescriptions. Outputs use standard medical code systems.
Tests LLMs (large language models) on 17 clinical datasets, 11 existing and 6 new. New tasks include open-ended clinical decisions, long documents, and new drugs.
Tests LLMs (large language models) on 5,000 multi-turn health conversations. Responses are graded against conversation-specific rubrics that 262 physicians wrote.
Tests LLMs (large language models) on real clinician chats with ChatGPT for care consults, documentation, and medical research. Physicians write and check the scoring rubrics.
Tests LLM agents in a virtual EHR (electronic health record) that uses the FHIR (Fast Healthcare Interoperability Resources) API. Agents do 300 physician-written tasks for 100 patients.
Tests whether LLMs (large language models) can do medical calculations from a patient note, such as risk scores and drug doses. Covers 55 calculator types.
A framework that tests LLMs (large language models) on medical tasks. It uses a clinician-checked list of 121 tasks and 35 benchmarks, some public and some gated or private.
Combines six open medical QA (question answering) datasets with HealthSearchQA, a new set of consumer medical questions. Adds human review of answers for harm and bias.
Tests speech models on finding clinical voice biomarkers, such as signs of Parkinson's disease, voice disorders, and depression. Uses five corpora in English, Spanish, German, and Chinese.
Best for
Testing speech models that screen for disease from voice recordings across languages.
Tests AI agents on long US healthcare workflows: prior authorization, utilization review, and care management. Agents use 20 simulated apps and a 1,290+ document policy handbook.
Best for
Evaluating clinical operations agents beyond medical question answering.