MetaStructAtlas：用于全身PET/CT中功能与结构推理的基于接地的3D视觉-语言数据集与基准

中文摘要 AI 辅助

代谢功能与解剖结构的联合解读对全身PET/CT的临床诊断至关重要。尽管近期3D医学视觉-语言模型取得了显著进展，但现有研究仅局限于区域CT成像，在全面的全身PET/CT分析领域存在关键空白。本研究中，我们引入MetaStructAtlas——一个用于接地全身PET/CT解读的大规模数据集，该数据集将多模态成像与整合的解剖、代谢及语义注释进行合成。MetaStructAtlas提供490个共配准的3D PET和CT体积，以及50470个器官级分割掩码和接地放射学报告。为促进交互式推理，我们进一步开发了MetaStructVQA——一个标准化的3D接地视觉问答基准，包含100565个问答对。该框架将诊断查询明确链接至多模态间的视觉证据，涵盖解剖、形态和代谢特征。最后，我们在MetaStructVQA上评估了最先进的3D医学VLMs，为核医学中的多模态表示学习和整合全身推理奠定了坚实基础。

英文摘要

The joint interpretation of metabolic function and anatomical structure is essential for clinical diagnosis in whole-body PET/CT. Although recent advances in 3D medical vision-language models have demonstrated remarkable progress, current efforts are limited to regional CT imaging, leaving a critical void in comprehensive whole-body PET/CT analysis. In this work, we introduce MetaStructAtlas, a large-scale dataset for grounded whole-body PET/CT interpretation that synthesizes multimodal imaging with integrated anatomical, metabolic, and semantic annotations. MetaStructAtlas provides 490 co-registered 3D PET and CT volumes with 50,470 organ-level segmentation masks and grounded radiology reports. To facilitate interactive reasoning, we further developed MetaStructVQA, a standardized 3D grounded visual question-answering benchmark containing 100,565 QA pairs. This framework explicitly links diagnostic queries to visual evidence across modalities, encompassing anatomical, morphological, and metabolic characteristics. Finally, we evaluate state-of-the-art 3D medical VLMs on MetaStructVQA, establishing a robust foundation for multimodal representation learning and integrated whole-body reasoning in nuclear medicine.

发表机构

• Shanghai Academy of Artificial Intelligence for Science（上海科学人工智能研究院）
• Artificial Intelligence Innovation and Incubation Institute, Fudan University（复旦大学人工智能创新与孵化研究院）
• Huashan Hospital（华山医院）
• Human Phenome Institute, Fudan University（复旦大学人类表型组研究院）
• Shanghai Universal Medical Imaging Diagnostic Center（上海全景医学影像诊断中心）

机构由 AI 辅助整理，请以论文原文为准。