Multimodal and voice evals add perception, timing, and interaction to language-model evaluation. For voice agents, end-to-end latency and recovery from errors often matter as much as accuracy.
How should multimodal and voice agents be evaluated?
Measure perception and task completion separately. Then add end-to-end checks for latency, interruptions, turn taking, tool use, background noise, accessibility, and recovery from misheard input.
18
Benchmarks in this domain
Sources first · alphabetical collection order
Runnability describes access to a usable repository, dataset, or harness—not whether setup is easy.
EVA-Bench
watchpartial
2026-10-26
Tests
Tests voice agents in simulated bot-to-bot audio conversations across three enterprise domains. Also tests how accents and background noise change results.
Gives vision-language models about 3,000 multiple-choice questions across 20 ability dimensions in English and Chinese. Uses CircularEval, which shuffles options to check consistent answers.
Gives multimodal models college-level questions from 30 subjects with images such as charts, maps, and diagrams. MMMU-Pro removes text-only answerable questions and adds a vision-only setting.
Compares speech-to-text, or ASR (automatic speech recognition), systems on public test sets. Covers English short-form, English long-form, and multilingual tracks.
Best for
Shortlisting open speech-recognition models with source-linked results.
Tests referring expression segmentation: the model gets an image and a phrase and must output pixel masks. Five levels isolate skills such as attributes, reading text, and spatial relations.
Best for
Specialist comparison of multimodal grounding and segmentation systems.
Tests GUI (graphical user interface) grounding: the model must locate a described element on a high-resolution screenshot. Covers 23 professional applications on three operating systems.
Best for
Choosing vision-language models for computer-use agents.
Tests Hindi ASR (automatic speech recognition) on held-out recordings from many Indian districts. Each segment has three independent human transcriptions.
Best for
Selecting ASR systems for Hindi-language products.
Asks multimodal models questions about videos, including groups of linked questions that test consistent reasoning. It is a new 2026 set, separate from Video-MME.
Best for
Comparing current video models on a set that is newer than Video-MME.
Tests VLA (vision-language-action) models on language-conditioned robot manipulation in simulation. Has 100 task categories, including long-horizon tasks that need common sense and reasoning.
Best for
Comparing embodied models on reproducible task primitives.
Tests speech interaction models on spoken conversations in English and Mandarin. Covers semantic quality, acoustic quality, chat ability, and performance under noise.
Tests speech language models on spoken agentic tasks: single tool calls, multi-tool workflows, multi-turn dialogue, and safety cases. Covers English and six Indic languages.
Tests LLM-based voice assistants with spoken instructions. Covers open QA (question answering), multiple-choice QA, reasoning, instruction following, safety, and variations in speaker, environment, and content.
Tests interactive video world models over multi-turn interactions such as navigation and event editing. Scores five dimensions: video quality, setting adherence, interaction adherence, consistency, and physics.
Best for
Comparing action-conditioned world models rather than passive video QA.