迷失在喧嚣中：揭示并剖析医学多模态大语言模型在图像分类中的性能退化
Lost in the Hype: Revealing and Dissecting the Performance Degradation of Medical Multimodal Large Language Models in Image Classification
AI总结：
本文研究了医学多模态大语言模型在图像分类中的性能退化问题，通过实验和特征探查揭示了四个失败模式，并提出了量化评分用于评估特征演变的健康状况。
摘要：
多模态大语言模型（MLLMs）的兴起在医学影像分析领域引发了前所未有的应用热潮。然而，作为最早集成到这一范式中的基础任务，医学图像分类揭示了一个令人沮丧的现实：最先进的医学MLLMs在预训练数据和模型参数上具有显著优势，却始终比传统深度学习模型表现更差。这种悖论促使我们重新思考：性能退化究竟源于何处？本文在三个代表性图像分类数据集上对14个开源医学MLLMs进行了广泛实验。超越表面的性能基准测试，我们采用特征探查技术追踪视觉特征模块和层的的信息流，使能够明确可视化分类信号的扭曲、稀释或覆盖之处。作为首次尝试剖析医学MLLMs分类性能退化问题，我们的发现揭示了四个失败模式：1）视觉表征的质量限制，2）连接器投影的保真度损失，3）LLM推理的理解不足，4）语义映射的错位。同时，我们引入了量化评分来表征特征演变的健康状况，使能够跨多样化的MLLMs和数据集进行原则性比较。此外，我们围绕当前医学MLLMs未能实现其承诺的临床潜力的关键障碍提供了深入讨论。我们希望我们的工作能引发社区的重新思考，强调从高期望到临床可部署的MLLMs的道路仍然漫长而曲折。
英文摘要：
The rise of multimodal large language models (MLLMs) has sparked an unprecedented wave of applications in the field of medical imaging analysis. However, as one of the earliest and most fundamental tasks integrated into this paradigm, medical image classification reveals a sobering reality: state-of-the-art medical MLLMs consistently underperform compared to traditional deep learning models, despite their overwhelming advantages in pre-training data and model parameters. This paradox prompts a critical rethinking: where exactly does the performance degradation originate? In this paper, we conduct extensive experiments on 14 open-source medical MLLMs across three representative image classification datasets. Moving beyond superficial performance benchmarking, we employ feature probing to track the information flow of visual features module-by-module and layer-by-layer throughout the entire MLLM pipeline, enabling explicit visualization of where and how classification signals are distorted, diluted, or overridden. As the first attempt to dissect classification performance degradation in medical MLLMs, our findings reveal four failure modes: 1) quality limitation in visual representation, 2) fidelity loss in connector projection, 3) comprehension deficit in LLM reasoning, and 4) misalignment of semantic mapping. Meanwhile, we introduce quantitative scores that characterize the healthiness of feature evolution, enabling principled comparisons across diverse MLLMs and datasets. Furthermore, we provide insightful discussions centered on the critical barriers that prevent current medical MLLMs from fulfilling their promised clinical potential. We hope that our work provokes rethinking within the community-highlighting that the road from high expectations to clinically deployable MLLMs remains long and winding.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
