TemMed-Bench：评估视觉语言模型的时序医学图像推理能力
TemMed-Bench: Evaluating Temporal Medical Image Reasoning in Vision-Language Models
AI总结：
本文提出TemMed-Bench时序医学图像推理基准，评估12个视觉语言模型，发现多数模型缺乏分析患者状况时序变化的能力，多模态检索扩充可显著提升其性能。
摘要：
现有的针对视觉语言模型的医学推理基准主要聚焦于基于单次就诊的图像分析患者状况，然而该设置与现实临床实践存在显著偏差——现实中医生通常会参考患者的历史状况，通过追踪其随时间的变化来做出全面评估。本文提出了TEMMED-BENCH这一多任务基准，旨在分析患者在不同临床就诊间的状况变化，以挑战大型视觉语言模型（LVLMs）的时序医学图像推理能力。TEMMED-BENCH的测试集包含视觉问答（VQA）、报告生成、图像对选择三项任务，以及一个包含超过17000个实例的补充知识语料库。我们利用TEMMED-BENCH对12个LVLMs进行了评估，其中包括6个专有模型和6个开源模型。结果显示，大多数LVLMs缺乏分析时序医学图像中患者状况变化的能力，在闭卷设置下，很大比例的模型仅达到随机猜测的水平。为了增强对状况变化的追踪，我们探索了在医学领域中，通过检索到的视觉和文本模态扩充输入的方法。我们还发现，与不进行检索或仅进行文本检索相比，多模态检索扩充在我们的基准上对大多数模型的性能提升显著，其中VQA任务的平均提升为2.59%。总体而言，我们构建了一个基于现实临床实践的基准，该基准揭示了LVLMs在时序医学图像推理方面的局限性，并强调多模态检索扩充是解决这一挑战的一个潜在有前景的研究方向。
英文摘要：
Existing medical reasoning benchmarks for vision-language models primarily focus on analyzing a patient's condition based on an image from a single visit. However, this setting deviates significantly from real-world clinical practice, where doctors typically refer to a patient's historical conditions to provide a comprehensive assessment by tracking their changes over time. In this paper, we introduce TEMMED-BENCH, a multi-task benchmark designed for analyzing changes in patients' conditions between different clinical visits, which challenges large vision-language models (LVLMs) to reason over temporal medical images. TEMMED-BENCH consists of a test set comprising three tasks - visual question-answering (VQA), report generation, and image-pair selection - and a supplementary knowledge corpus of over 17,000 instances. With TEMMED-BENCH, we conduct an evaluation of twelve LVLMs, comprising six proprietary and six open-source models. Our results show that most LVLMs lack the ability to analyze patients' condition changes over temporal medical images, and a large proportion perform only at a random-guessing level in the closed-book setting. To enhance the tracking of condition changes, we explore augmenting the input with both retrieved visual and textual modalities in the medical domain. We also show that multi-modal retrieval augmentation yields notably higher performance gains than no retrieval and textual retrieval alone across most models on our benchmark, with the VQA task showing an average improvement of 2.59%. Overall, we compose a benchmark grounded on real-world clinical practice, and it reveals LVLMs' limitations in temporal medical image reasoning, as well as highlighting the use of multi-modal retrieval augmentation as a potentially promising direction worth exploring to address this challenge.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
