本研究开发了一种自动化弱标签流程生成PET/CT图像-文本对数据集，并训练了3D视觉定位模型ConTEXTual Net 3D，其性能显著优于其他模型但不及核医学医师，表明需要更大数据集以缩小差距。
摘要：
视觉-语言模型可以通过视觉定位将物体的文本描述与其在图像中的具体位置联系起来。这在增强放射学报告中具有潜在应用。然而，这些模型需要大型标注图像-文本数据集，而PET/CT缺乏此类数据。我们开发了一个自动化流程，用于生成将PET/CT报告描述与其图像位置关联的弱标签，并利用它训练了一个3D视觉-语言视觉定位模型。我们的流程通过识别SUVmax和轴向切片编号的提及，在PET/CT报告中寻找阳性发现。从25,578例PET/CT检查中，我们提取了11,356个句子-标签对。使用这些数据，我们训练了ConTEXTual Net 3D，它通过token级别的交叉注意力将来自大型语言模型的文本嵌入与3D nnU-Net集成在一起。该模型的性能与LLMSeg、ConTEXTual Net的2.5D版本以及两名核医学医师进行了比较。弱标签流程在98%的病例（246/251）中准确识别了病灶位置，其中7.5%需要边界调整。ConTEXTual Net 3D的F1分数为0.80，优于LLMSeg（F1=0.22）和2.5D模型（F1=0.53），尽管其表现不及两名医师（F1分别为0.94和0.91）。该模型在FDG（F1=0.78）和DCFPyL（F1=0.75）检查中表现更好，而在DOTATE（F1=0.58）和Fluciclovine（F1=0.66）上表现下降。该模型在不同病灶大小上表现一致，但对低摄取病灶的准确性降低。我们新颖的弱标签流程准确生成了PET/CT图像-文本对的标注数据集，促进了3D视觉定位模型的发展。ConTEXTual Net 3D显著优于其他模型，但未达到核医学医师的表现水平。我们的研究表明，可能需要更大的数据集来缩小这一性能差距。
英文摘要：
Vision-language models can connect the text description of an object to its specific location in an image through visual grounding. This has potential applications in enhanced radiology reporting. However, these models require large annotated image-text datasets, which are lacking for PET/CT. We developed an automated pipeline to generate weak labels linking PET/CT report descriptions to their image locations and used it to train a 3D vision-language visual grounding model. Our pipeline finds positive findings in PET/CT reports by identifying mentions of SUVmax and axial slice numbers. From 25,578 PET/CT exams, we extracted 11,356 sentence-label pairs. Using this data, we trained ConTEXTual Net 3D, which integrates text embeddings from a large language model with a 3D nnU-Net via token-level cross-attention. The model's performance was compared against LLMSeg, a 2.5D version of ConTEXTual Net, and two nuclear medicine physicians. The weak-labeling pipeline accurately identified lesion locations in 98% of cases (246/251), with 7.5% requiring boundary adjustments. ConTEXTual Net 3D achieved an F1 score of 0.80, outperforming LLMSeg (F1=0.22) and the 2.5D model (F1=0.53), though it underperformed both physicians (F1=0.94 and 0.91). The model achieved better performance on FDG (F1=0.78) and DCFPyL (F1=0.75) exams, while performance dropped on DOTATE (F1=0.58) and Fluciclovine (F1=0.66). The model performed consistently across lesion sizes but showed reduced accuracy on lesions with low uptake. Our novel weak labeling pipeline accurately produced an annotated dataset of PET/CT image-text pairs, facilitating the development of 3D visual grounding models. ConTEXTual Net 3D significantly outperformed other models but fell short of the performance of nuclear medicine physicians. Our study suggests that even larger datasets may be needed to close this performance gap.
Enhancing Abnormality Grounding for Vision Language Models with Knowledge Descriptions
Technical University of Munich（慕尼黑工业大学）；Munich Center for Machine Learning（慕尼黑机器学习中心）；Helmholtz AI（亥姆霍兹人工智能研究所）；Helmholtz Munich（亥姆霍兹慕尼黑中心）；Imperial College London（伦敦帝国学院）；King’s College London（伦敦国王学院）
CPT: Colorful Prompt Tuning for Pre-trained Vision-Language Models
Tsinghua University（清华大学）；Beijing National Research Center for Information Science and Technology（北京信息科学与技术国家研究中心）；National University of Singapore（新加坡国立大学）
TransVG++: End-to-End Visual Grounding with Language Conditioned Vision Transformer
University of Science and Technology of China（中国科学技术大学）；Institute of Artificial Intelligence, Hefei Comprehensive National Science Center（合肥综合性国家科学中心人工智能研究院）；Microsoft（微软）；JD Explore Academy（京东探索研究院）；Amazon（亚马逊）；University of Sydney（悉尼大学）；Shanghai Artificial Intelligent Laboratory（上海人工智能实验室）
A Unified Vision-Language Model for PSMA PET/CT Report Generation, Visual Question Answering, and Lesion Segmentation
University of Florida（佛罗里达大学）；University of California, San Francisco（加利福尼亚大学旧金山分校）；Stony Brook University（纽约州立大学石溪分校）；The University of Texas MD Anderson Cancer Center（得克萨斯大学MD安德森癌症中心）
From Region Arrival to Instance-Level Grounding in Vision-and-Language Navigation
Australian Institute for Machine Learning, Adelaide University（澳大利亚机器学习研究所，阿德莱德大学）；Ruyi Dynamics（如意动力）；Fudan University（复旦大学）；Swiss Federal Institute of Technology Lausanne (EPFL)（瑞士洛桑联邦理工学院）
DDVT: Dynamic Dual-level Vision Transformer Fusion Network for Answer Grounding in Visual Question Answering
National and Local Joint Engineering Laboratory of Computer Aided Design, School of Software Engineering, Dalian University（大连大学软件工程学院计算机辅助设计国家地方联合工程实验室）；School of Computer Science and Technology, Dalian University of Technology（大连理工大学计算机科学与技术学院）
GeoGround: A Unified Large Vision-Language Model for Remote Sensing Visual Grounding
Nanyang Technological University（南洋理工大学）；University of Reading（阅读大学）；Shanghai Jiao Tong University（上海交通大学）；Harbin Institute of Technology（哈尔滨工业大学）；SenseTime Research（商汤科技研究院）
PETAR: Localized Findings Generation with Mask-Aware Vision-Language Modeling for PET Automated Reporting
University of Wisconsin–Madison Department of Computer Sciences（威斯康星大学麦迪逊分校计算机科学系）；University of Wisconsin–Madison Department Radiology（威斯康星大学麦迪逊分校放射学系）；Microsoft（微软公司）
MIMO: A medical vision language model with visual referring multimodal input and pixel grounding multimodal output
School of Software & Microelectronics, Peking University（软件与微电子学院，北京大学）；School of Computer Science, Peking University（计算机学院，北京大学）；National Engineering Research Center for Software Engineering, Peking University（软件工程国家工程研究中心，北京大学）；Peking University Sixth Hospital（北京大学第六医院）；Augusta University（奥古斯塔大学）；Peking University First Hospital（北京大学第一医院）
Detecting Clinical Hallucinations in LVLMs via Counterfactual Visual Grounding Uncertainty
School of Intelligent Science and Technology, Nanjing University（南京大学智能科学与技术学院）；National Institute of Healthcare Data Science at Nanjing University（南京大学医疗数据科学国家研究院）；School of Biomedical Engineering, Nanjing University（南京大学生物医学工程学院）
Do Medical Vision Language Models Actually See? A Counterfactual Grounding Framework and Hard-Negative Contrastive Training for Visually-Reliant Medical VLMs
The University of Texas MD Anderson Cancer Center（德克萨斯大学MD安德森癌症中心）；Eisai Inc.（卫材株式会社）；IISc, Bangalore（印度科学研究所班加罗尔分校）；Cohere Labs Community（Cohere实验室社区）
EPIC-Bench: A Perception-Centric Benchmark for Fine-Grained Embodied Visual Grounding in Vision-Language Models
X-Humanoid；Fudan University（复旦大学）；University of Science and Technology of China（中国科学技术大学）；University of Manchester（曼彻斯特大学）；Monash University（墨尔本大学）；Celonis AI；University of New South Wales（新南威尔士大学）
MetaStructAtlas: A Grounded 3D Vision-Language Dataset and Benchmark for Functional and Structural Reasoning in Whole-Body PET/CT
Shanghai Academy of Artificial Intelligence for Science（上海科学人工智能研究院）；Artificial Intelligence Innovation and Incubation Institute, Fudan University（复旦大学人工智能创新与孵化研究院）；Huashan Hospital（华山医院）；Human Phenome Institute, Fudan University（复旦大学人类表型组研究院）；Shanghai Universal Medical Imaging Diagnostic Center（上海全景医学影像诊断中心）
