arXiv 2601.027372026-01-15cs.CV

揭示和弥合MLLMs中的功能感知差距：通过PET-Bench实现原子视觉对齐和分层评估

Unveiling and Bridging the Functional Perception Gap in MLLMs: Atomic Visual Alignment and Hierarchical Evaluation via PET-Bench

School of Biomedical Engineering, Southern Medical University（生物医学工程学院，南方医科大学）；School of Biomedical Engineering, Shanghai Jiaotong University（生物医学工程学院，上海交通大学）；Department of Electronic Engineering, Chinese University of Hong Kong（电子工程系，中国香港大学）；Faculty of Dentistry, The University of Hong Kong（牙科学院，香港大学）；Department of Nuclear Medicine, The Second Affiliated Hospital of Guangzhou University of Chinese Medicine（核医学科，广州中医药大学第二附属医院）；PET Center, Department of Nuclear Medicine, Guangdong Provincial People’s Hospital, Southern Medical University（PET中心，核医学科，广东省人民医院，南方医科大学）；Department of Nuclear Medicine, Nanfang Hospital, Southern Medical University（核医学科，南芳医院，南方医科大学）；Division of Nuclear Medicine and Molecular Imaging, Geneva University Hospitals（核医学与分子影像学部，日内瓦大学医院）；Departments of Radiology, Physics, and Biomedical Engineering, The University of British Columbia（放射学、物理和生物医学工程系，不列颠哥伦比亚大学）；Medical Artificial Intelligence Laboratory, Westlake University（医学人工智能实验室，西湖大学）

Zanting Ye, Xiaolong Niu, Xuanbin Wu, Xu Han, Shengyuan Liu, Jing Hao, Zhihao Peng, Hao Sun, Jieqin Lv, Fanghu Wang, Yanchao Huang, Hubing Wu, Yixuan Yuan, Habi… 展开作者

Zanting Ye, Xiaolong Niu, Xuanbin Wu, Xu Han, Shengyuan Liu, Jing Hao, Zhihao Peng, Hao Sun, Jieqin Lv, Fanghu Wang, Yanchao Huang, Hubing Wu, Yixuan Yuan, Habib Zaidi, Arman Rahmim, Yefeng Zheng, Lijun Lu

备注：: 9 pages, 6 figures, 6 tables

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

本文提出AVA方法，通过原子视觉对齐解决MLLMs在功能成像中的感知差距，提升诊断准确性14.83%。

摘要：

尽管多模态大语言模型（MLLMs）在异常检测和解剖模态报告生成等任务中表现出色，但其在功能成像方面的能力仍鲜为人知。在本工作中，我们识别并量化了一个基本的功能感知差距：当前视觉编码器无法独立于形态学先验解码功能示踪剂的生物分布。将正电子发射断层扫描（PET）视为研究这一断层的关键模态，我们引入了PET-Bench，这是首个大规模的功能成像基准，包含来自9732个多中心、多示踪剂PET研究的52308个分层问答对。对19种最先进的MLLMs进行广泛评估，发现了一个关键的安全隐患，即思维链（CoT）幻觉陷阱。我们观察到，标准的CoT提示，尽管被认为能增强推理，却反而在PET中将语言生成与视觉证据脱节，产生临床流畅但事实不成立的诊断。为了解决这一问题，我们提出了原子视觉对齐（AVA），一种简单的微调策略，要求在高阶诊断推理之前掌握低阶功能感知。我们的结果表明，AVA有效弥合了感知差距，将CoT从幻觉源转变为稳健的推理工具，并将诊断准确性提高了14.83%。代码和数据可在https://github.com/yezanting/PET-Bench获取。

英文摘要：

While Multimodal Large Language Models (MLLMs) have demonstrated remarkable proficiency in tasks such as abnormality detection and report generation for anatomical modalities, their capability in functional imaging remains largely unexplored. In this work, we identify and quantify a fundamental functional perception gap: the inability of current vision encoders to decode functional tracer biodistribution independent of morphological priors. Identifying Positron Emission Tomography (PET) as the quintessential modality to investigate this disconnect, we introduce PET-Bench, the first large-scale functional imaging benchmark comprising 52,308 hierarchical QA pairs from 9,732 multi-site, multi-tracer PET studies. Extensive evaluation of 19 state-of-the-art MLLMs reveals a critical safety hazard termed the Chain-of-Thought (CoT) hallucination trap. We observe that standard CoT prompting, widely considered to enhance reasoning, paradoxically decouples linguistic generation from visual evidence in PET, producing clinically fluent but factually ungrounded diagnoses. To resolve this, we propose Atomic Visual Alignment (AVA), a simple fine-tuning strategy that enforces the mastery of low-level functional perception prior to high-level diagnostic reasoning. Our results demonstrate that AVA effectively bridges the perception gap, transforming CoT from a source of hallucination into a robust inference tool and improving diagnostic accuracy by up to 14.83%. Code and data are available at https://github.com/yezanting/PET-Bench.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。