G2D: From Global to Dense Radiography Representation Learning via Vision-Language Pre-training
Imperial College London(帝国理工学院);University of Oxford(牛津大学);École des Ponts(巴黎桥路学校);EDF R&D(法国电力公司研发中心);Royal Brompton and Harefield Hospitals(皇家布朗普顿和 Harefield 医院)
近年来,医学视觉-语言预训练(VLP)在从医学图像及其配对放射学报告中学习全局视觉表征方面取得了实质性进展。然而,现实世界中的医学影像任务通常需要更细粒度的视觉特征。这些任务包括视觉定位任务(例如语义分割、目标检测)以及视觉指代定位任务。但是,当前的医学VLP方法在学习这些细粒度特征方面面临挑战,因为它们主要关注图像块与单个文本token之间的暴力对齐以进行局部视觉特征学习,这对于下游密集预测任务而言并非最优。在本研究中,我们提出了一个名为Global to Dense level representation learning(G2D)的新型VLP框架,与现有医学VLP方法相比,该框架显著提升了所学特征的粒度并实现了更准确的指代定位。具体而言,G2D通过与全局视觉-语言对齐并行的伪分割任务,学习密集且具有语义根基的图像表征。值得注意的是,生成伪分割目标不会产生额外的可训练参数:这些目标是在VLP过程中通过无参数处理器即时获得的。G2D在6项医学影像任务和25种疾病上均取得了优越性能,尤其是在需要细粒度、语义根基图像特征的语义分割任务中。在该任务上,即使仅使用1%的训练数据进行微调,而对比模型使用100%的训练数据,G2D仍优于这些同类模型。代码可在https://github.com/cheliu-computation/G2D-NeurIPS24/tree/main获取。
英文摘要:
Recently, medical vision-language pre-training (VLP) has reached substantial progress to learn global visual representation from medical images and their paired radiology reports. However, medical imaging tasks in real world usually require finer granularity in visual features. These tasks include visual localization tasks (e.g., semantic segmentation, object detection) and visual grounding task. Yet, current medical VLP methods face challenges in learning these fine-grained features, as they primarily focus on brute-force alignment between image patches and individual text tokens for local visual feature learning, which is suboptimal for downstream dense prediction tasks. In this work, we propose a new VLP framework, named \textbf{G}lobal to \textbf{D}ense level representation learning (G2D) that achieves significantly improved granularity and more accurate grounding for the learned features, compared to existing medical VLP approaches. In particular, G2D learns dense and semantically-grounded image representations via a pseudo segmentation task parallel with the global vision-language alignment. Notably, generating pseudo segmentation targets does not incur extra trainable parameters: they are obtained on the fly during VLP with a parameter-free processor. G2D achieves superior performance across 6 medical imaging tasks and 25 diseases, particularly in semantic segmentation, which necessitates fine-grained, semantically-grounded image features. In this task, G2D surpasses peer models even when fine-tuned with just 1\% of the training data, compared to the 100\% used by these models. The code can be found in https://github.com/cheliu-computation/G2D-NeurIPS24/tree/main.
Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutions
Nanjing University(南京大学);The University of Hong Kong(香港大学);Nanjing University of Science and Technology(南京理工大学);IIAI(智能信息处理与应用研究院);SenseTime Research(商汤科技研究院)
CLIPSelf: Vision Transformer Distills Itself for Open-Vocabulary Dense Prediction
Nanyang Technological University(南洋理工大学);The Chinese University of Hong Kong(香港中文大学);The University of Hong Kong(香港大学);SenseTime Research and Tetras.AI(商汤科技研究院与Tetras.AI);Shanghai AI Laboratory(上海人工智能实验室)
针对 CLIP ViT 从全局图像迁移到局部区域时的域偏移问题,本文提出 CLIPSelf 自蒸馏方法,在无需区域-文本对的情况下增强区域-语言对齐,并在多项开放词汇密集预测基准上达到最优性能。
DenseMLLM: Standard Multimodal LLMs for Dense Prediction
Department of Electronic and Computer Engineering, The Hong Kong University of Science and Technology, Hong Kong, China(香港科技大学电子与计算机工程系);Tencent, Youtu-Lab, China(腾讯优图实验室)
Vision-language models for chest radiography do not always need the image
Pattern Recognition Lab, Friedrich-Alexander-Universität Erlangen-Nürnberg(弗里德里希-亚历山大-埃尔朗根-纽伦堡大学模式识别实验室);Department of Diagnostic and Interventional Radiology, TUM University Clinic, School of Medicine and Health, Klinikum rechts der Isar, Technical University of Munich(慕尼黑工业大学医学院与健康学院伊萨尔河右岸医院诊断与介入放射学系);Lab for AI in Medicine, RWTH Aachen University(亚琛工业大学医学人工智能实验室);Department of Diagnostic and Interventional Radiology, University Hospital RWTH Aachen(亚琛工业大学医院诊断与介入放射学系)
VPNeXt -- Rethinking Dense Decoding for Plain Vision Transformer
School of Information and Software Engineering, University of Electronic Science and Technology of China(信息与软件工程学院,电子科学与技术大学);Shenzhen Institute for Advanced Study, University of Electronic Science and Technology of China(深圳先进研究所,电子科学与技术大学)
Less Could Be Better: Parameter-efficient Fine-tuning Advances Medical Vision Foundation Models
School of Informatics, Xiamen University(厦门大学信息学院);Department of Biomedical Informatics, Harvard University(哈佛大学生物医学信息学系);Department of Computer Science, The University of Hong Kong(香港大学计算机科学系)
本文实证评估 LoRA 对胸部 X 线摄影自监督基础模型的微调效果,证明其以不足 1% 参数在多数任务上超过全参数微调,并刷新数据高效学习基准。
Enhancing Representation in Radiography-Reports Foundation Model: A Granular Alignment Algorithm Using Masked Contrastive Learning
Paul C. Lauterbur Research Center for Biomedical Imaging(保罗·C·劳特伯生物医学成像研究中心);Shenzhen Institute of Advanced Technology(深圳先进技术研究院);Pengcheng Laboratory(鹏城实验室);University of Chinese Academy of Sciences(中国科学院大学);Harvard University(哈佛大学);Shanghai AI Lab(上海人工智能实验室)
A Generative Foundation Model for Chest Radiography
Department of Radiation Oncology, Stanford University School of Medicine(斯坦福大学医学院放射肿瘤科);Department of Computer Science, School of Computing and Data Science, The University of Hong Kong(香港大学计算机科学系);Department of Population Medicine and Diagnostic Sciences, College of Veterinary Medicine, Cornell University(康奈尔大学兽医学院人口医学与诊断科学系);Department of Radiology, The First Affiliated Hospital of Jinan University(暨南大学第一附属医院放射科);Department of Radiology, Stanford University School of Medicine(斯坦福大学医学院放射科);Department of Computer Science and Engineering, The Chinese University of Hong Kong(香港中文大学计算机科学与工程系);Department of Biomedical Informatics, Harvard Medical School(哈佛医学院生物医学信息学系)
XBench: A Comprehensive Benchmark for Visual-Language Explanations in Chest Radiography
ARTORG Center for Biomedical Engineering Research, University of Bern, Switzerland(ARTORG生物医学工程研究中心,伯尔尼大学,瑞士);Shanghai Jiao Tong University, China(上海交通大学,中国);Kaiko.AI, Switzerland(Kaiko.AI,瑞士);Dept. of Radiation Oncology, Inselspital, Bern University Hospital(放射肿瘤科,因斯普尔茨医院,伯尔尼大学医院)
Frozen Foundation-Model Embeddings Discard Small-Lesion Signal in Chest Radiography: Implications for Pre-Deployment Evaluation
Department of Biomedical Engineering and Informatics, Indiana University(印第安纳大学生物医学工程与信息学系);Department of Radiology and Imaging Sciences, Emory University(埃默里大学放射学与影像科学系)
DensePercept-NCSSD: Vision Mamba towards Real-time Dense Visual Perception with Non-Causal State Space Duality
Machine Intelligence Group, Department of Computer Science & Information Systems, Birla Institute of Technology and Science, Pilani, Hyderabad Campus(机器智能组,计算机科学与信息系统系,比拉理工学院,帕利尼,海得拉巴校区)
Self-supervised vision-langage alignment of deep learning representations for bone X-rays analysis
Information and Communication Technologies, Electronics and Applied Mathematics (ICTEAM)(信息、通信技术、电子与应用数学研究所(ICTEAM));UCLouvain(天主教鲁汶大学);Service de chirurgie orthopédique et traumatologie, cliniques universitaires Saint-Luc(圣卢克大学医院骨科与创伤外科);Institut de Recherche Expérimentale et Clinique (IREC)(实验与临床研究所(IREC))
On the Importance of Accurate Geometry Data for Dense 3D Vision Tasks
Technical University of Munich(慕尼黑工业大学);Huawei Noah’s Ark Lab(华为诺亚方舟实验室);Siemens AG(西门子股份公司)
解决密集3D视觉问题的基于学习的方法通常在3D传感器数据上进行训练。分别使用的测量距离的原理提供了优点和缺点。由于缺乏多模态数据集,这些通常不会在文献中进行比较或讨论。无纹理区域对于来自运动运动恢复结构和立体声的结构是有问题的,反射材料对主动感测造成问题,并且半透明对象的距离对于用现有硬件测量是复杂的。对不准确或损坏的数据进行训练会导致模型偏差,并妨碍泛化能力。如果在评估过程中将传感器测量值视为地面实况,则这些影响仍然不会被注意到。本文研究了传感器误差对密集三维视觉任务深度估计和重建的影响。我们严格显示传感器特性对学习预测的重大影响,并注意到日常家庭环境中各种技术所产生的泛化问题。为了进行评估,我们引入了精心设计的数据集\footnote{dataset available at https://github.com/Junggy/HAMMER-dataset},其中包括来自商用传感器的测量结果,即D-ToF、I-ToF、无源/有源立体声和单目RGB+ P。我们的研究量化了相当大的传感器噪声影响,并为改善密集视觉估计和有针对性的数据融合铺平了道路。