*Equal contribution
Vision and language models (VLMs) hold immense promise to transform biomedical imaging workflows, from detecting lesions in chest X-rays to profiling cellular features in microscopy. Realizing this potential, however, requires robust and fine-grained visual perception. Models need to correctly interpret subtle features in images, and they must do so across diverse biomedical modalities, scales, and contexts. Nevertheless, current benchmarks remain limited. To address these gaps, we introduce the Massive Multimodal Biomedical Understanding (MMBU) benchmark. It is the largest biomedical vision and language benchmark to date, covering 35 submodalities with rich structured metadata. It includes both open and closed versions of ungrounded classification, grounded classification, and object detection, enabling systematic evaluation of model performance across biological scales, clinical settings, and imaging modalities. Evaluating 15 open-weight and 2 frontier VLMs, we find that while medical adaptation provides measurable gains for some models, the high accuracy often reported on established benchmarks can mask deficiencies in visual perception and domain generalization.
The MMBU dataset is currently under a 3-month embargo following publication. Full public release is planned after this period. In the meantime, stay tuned — an MMBU benchmark challenge is coming soon.
410 curated datasets spanning 11 modalities, 35 submodalities, 20 specimens, and 95 body-part regions — with 13 structured metadata fields per sample including provenance, modality, stain, specimen, and institution. Stats shown in Figure 2
Ungrounded classification, grounded classification (from segmentation masks and bounding boxes), and object detection — each in open- and closed-ended VQA formats with three context levels (no context, modality, full context).
17 VLMs including MedGemma, Lingshu, OctoMed, LLaVA-Med, InternVL3.5, Qwen2.5/3-VL, GPT-4.1-mini, and GPT-5.4-mini — plus their general-domain base counterparts where applicable.
Multidisciplinary expert taxonomy → dataset discovery across Zenodo, Kaggle, and Hugging Face → metadata standardization → metadata-driven question construction with human-in-the-loop validation.
We evaluated 17 state-of-the-art vision-language models (VLMs) across diverse biomedical imaging tasks and found that performance remains surprisingly limited. Although some models achieve strong results on individual benchmarks, no model consistently performs well across the full spectrum of biomedical perception tasks (Figure 3).
Across classification tasks, performance drops dramatically when models must generate answers freely rather than choose from predefined options (Figure 3). This large gap suggests that many current biomedical VLMs rely on answer-choice cues instead of robust visual understanding.
While several models can correctly classify findings when provided with a region of interest, they struggle to accurately localize those findings themselves (Figure 3). Detection performance remains near random across most models, highlighting significant limitations in spatial reasoning and visual grounding.
Different models excel on different tasks (Figure 3). GPT-based models perform strongly on some classification tasks, InternVL excels on certain grounded reasoning tasks, and Qwen-based models achieve the highest performance on segmentation-derived classification. However, no model consistently ranks first across all evaluation settings.
Models trained specifically on medical data not always outperfom their general-purpose counterparts (Figure 5A). The models that do (OctoMed and MedGemma), show modest gain. In many cases, medically adapted models perform similarly or worse to their base versions, suggesting that current adaptation strategies only partially address the challenges of biomedical visual understanding.
The success of medical adaptation appears to depend more on the amount of biomedical training data than on model size alone (Figure 5B). Models trained on larger medical datasets tend to show more consistent improvements across tasks.
Many specialized medical models achieve strong results on widely used benchmarks such as PathVQA, VQA-RAD, and SLAKE (Figure 6). However, these gains frequently fail to transfer to broader and more diverse biomedical settings, suggesting that current benchmarks may overestimate real-world capability.
Our results indicate that biomedical visual perception remains an open challenge. Despite rapid progress in multimodal AI, current models still struggle with robust visual understanding, grounded reasoning, and spatial localization across diverse biomedical domains. MMBU provides a comprehensive framework for measuring these limitations and guiding the development of more reliable biomedical AI systems.
@article{dcunha2026mmbu,
title = {MMBU: A Massive Multi-modal Biomedical Understanding Benchmark to Probe the Perception Capabilities of Vision-Language Models},
author = {D'Cunha, Ryan and Lozano, Alejandro and Sun, Xiaoxiao and Vela Jarquin, Daniel and Sun, Min Woo and Aklilu, Josiah and Burgess, James and Zhang, Yuhui and Nayebi, Ryan and Avila Robayo, Paola and Ye, Jin and Hu, Ming and Deng, Zhongying and He, Junjun and Chen, Xin and Yao, Yue and Tibshirani, Robert and Nirschl, Jeffrey J. and Yeung-Levy, Serena},
journal = {arXiv preprint arXiv:2606.06696},
year = {2026},
eprint = {2606.06696},
archivePrefix = {arXiv},
primaryClass = {cs.CV},
url = {https://arxiv.org/abs/2606.06696}
}