分析与缓解多模态医学视觉问答中的跨语言退化问题
Analyzing and Mitigating Cross-Lingual Degradation in Multilingual Medical VQA
浏览论文内容
中文总结 AI 辅助
针对多模态医学VQA中LVLMs的跨语言退化问题，构建多语言基准并提出MedVL-XLRepE方法，可在3种LVLMs和8种语言上将退化缓解幅度提升至最高6.33%。
中文摘要 AI 辅助
医学视觉问答（VQA）是临床人工智能领域的关键任务，但其评估迄今几乎仅聚焦于英语，限制了对语言多样化患者和临床医生的适用性。近期多模态医学VQA基准显示，大型视觉语言模型（LVLMs）在非英语语言中存在性能退化，但缺乏对跨语言差异如何影响医学VQA所需不同能力的细粒度分析。为此，我们构建了涵盖8种语言的多模态医学VQA基准，分为4个代表性场景，以隔离医学VQA所需的核心能力。通过评估5种开源和闭源LVLMs，我们发现跨语言退化并非均匀分布，而是高度依赖场景。因此，我们提出MedVL-XLRepE，一种无需训练的场景感知表示工程方法，利用LVLMs在英语医学VQA上的卓越能力，在推理时引导非英语表示向其英语对应表示靠拢。在3种LVLMs和8种语言上，MedVL-XLRepE可持续缓解跨语言退化，提升幅度最高达6.33%。
英文摘要
Medical visual question answering (VQA) is a crucial task in clinical AI, yet its evaluation has so far centered almost exclusively on English, limiting its relevance to linguistically diverse patients and clinicians. Recent multilingual medical VQA benchmarks show that large vision-language models (LVLMs) degrade in non-English languages, but lack a fine-grained analysis of how cross-lingual variation affects the distinct capabilities that medical VQA requires. To this end, we construct a multilingual medical VQA benchmark over eight languages, organized into four representative scenarios that isolate the core capabilities medical VQA requires. Evaluating five open- and closed-source LVLMs, we find that cross-lingual degradation is not uniform but highly scenario-dependent. We therefore propose MedVL-XLRepE, a training-free scenario-aware representation engineering method, leveraging LVLMs' superior English medical VQA capability to steer non-English representations toward their English counterparts at inference time. Across three LVLMs and eight languages, MedVL-XLRepE consistently mitigates cross-lingual degradation, with gains of up to 6.33\%.
发表机构
- Research Center for Social Computing and Interactive Robotics（社会计算与交互机器人研究中心）
- Harbin Institute of Technology（哈尔滨工业大学）
机构由 AI 辅助整理，请以论文原文为准。
