Learning to Exploit Temporal Structure for Biomedical Vision-Language Processing
Microsoft Health Futures（微软健康未来部门）
Shruthi Bannur, Stephanie Hyland, Qianchu Liu, Fernando Pérez-García, Maximilian Ilse, Daniel C. Castro, Benedikt Boecking, Harshita Sharma, Kenza Bouzid, Anja … 展开作者
Shruthi Bannur, Stephanie Hyland, Qianchu Liu, Fernando Pérez-García, Maximilian Ilse, Daniel C. Castro, Benedikt Boecking, Harshita Sharma, Kenza Bouzid, Anja Thieme, Anton Schwaighofer, Maria Wetscherek, Matthew P. Lungren, Aditya Nori, Javier Alvarez-Valle, Ozan Oktay
Self-supervised learning in vision-language processing exploits semantic alignment between imaging and text modalities. Prior work in biomedical VLP has mostly relied on the alignment of single image and report pairs even though clinical notes commonly refer to prior images. This does not only introduce poor alignment between the modalities but also a missed opportunity to exploit rich self-supervision through existing temporal content in the data. In this work, we explicitly account for prior images and reports when available during both training and fine-tuning. Our approach, named BioViL-T, uses a CNN-Transformer hybrid multi-image encoder trained jointly with a text model. It is designed to be versatile to arising challenges such as pose variations and missing input images across time. The resulting model excels on downstream tasks both in single- and multi-image setups, achieving state-of-the-art performance on (I) progression classification, (II) phrase grounding, and (III) report generation, whilst offering consistent improvements on disease classification and sentence-similarity tasks. We release a novel multi-modal temporal benchmark dataset, MS-CXR-T, to quantify the quality of vision-language representations in terms of temporal semantics. Our experimental results show the advantages of incorporating prior images and reports to make most use of the data.
BioVFM-21M: Benchmarking and Scaling Self-Supervised Vision Foundation Models for Biomedical Image Analysis
Paul C. Lauterbur Research Center for Biomedical Imaging（生物医学成像保罗·劳特伯研究中心）；Shenzhen Institutes of Advanced Technology（深圳先进技术研究院）；Chinese Academy of Sciences（中国科学院）；Pengcheng Laboratory（鹏城实验室）；University of Chinese Academy of Sciences（中国科学院大学）；Faculty of Applied Sciences（应用科学学院）
Unifying Biomedical Vision-Language Expertise: Towards a Generalist Foundation Model via Multi-CLIP Knowledge Distillation
Department of Radiation Oncology, Winship Cancer Institute, Emory University School of Medicine（放射肿瘤科，Winship癌症研究所，埃默里大学医学院）；Department of Biomedical Engineering, College of Engineering, Georgia Institute of Technology（生物医学工程系，工程学院，佐治亚理工学院）；Department of Computer Science and Mathematics, Laney Graduate School, Emory University（计算机科学与数学系，兰尼研究生院，埃默里大学）；School of Electrical and Computer Engineering, College of Engineering, Georgia Institute of Technology（电气与计算机工程系，工程学院，佐治亚理工学院）
From $\mathcal{O}(n^{2})$ to $\mathcal{O}(n)$ Parameters: Quantum Self-Attention in Vision Transformers for Biomedical Image Classification
UCL Hawkes Institute（UCL哈维斯研究所）；Department of Medical Physics and Biomedical Engineering（医学物理与生物医学工程系）；University College London（伦敦大学学院）；University College London Hospitals NHS Foundation Trust（伦敦大学学院医院国家健康服务信托基金会）；Human Physiology and Performance Laboratory (HPPL)（人体生理与表现实验室）；Centre for Peri-operative Medicine（围手术期医学中心）；Department of Targeted Intervention（精准干预系）；Division of Surgery and Interventional Science（外科与介入科学 division）
MELINDA: A Multimodal Dataset for Biomedical Experiment Method Classification
University of California, Los Angeles（加利福尼亚大学洛杉矶分校）；University of Southern California（南加利福尼亚大学）；Intuit Inc.（英图伊特公司）；Chan Zuckerberg Initiative（陈·扎克伯格倡议）
MedMNIST v2 -- A large-scale lightweight benchmark for 2D and 3D biomedical image classification
Shanghai Jiao Tong University（上海交通大学）；Boston College（波士顿学院）；RWTH Aachen University（亚琛工业大学）；Fudan University（复旦大学）；Shanghai General Hospital（上海市第一人民医院）；Shanghai Jiao Tong University School of Medicine（上海交通大学医学院）；Harvard University（哈佛大学）
From Panel to Pixel: Zoom-In Vision-Language Pretraining from Biomedical Scientific Literature
University of Strasbourg, CNRS, INSERM, ICube, UMR7357, Strasbourg, France（斯特拉斯堡大学、法国国家科学研究中心、法国国家医学研究院、ICube、UMR7357、斯特拉斯堡，法国）；Technical University of Munich（慕尼黑技术大学）；Stanford University（斯坦福大学）；DSAI, The Hong Kong Polytechnic University（香港理工大学DSAI实验室）；University of British Columbia（不列颠哥伦比亚大学）；Munich Center for Machine Learning（慕尼黑机器学习中心）；IHU Strasbourg, Strasbourg（斯特拉斯堡IHU医院，斯特拉斯堡）；Vector Institute for AI（人工智能向量研究所）；Yale University（耶鲁大学）
MMBU: A Massive Multi-modal Biomedical Understanding Benchmark to Probe the Perception Capabilities of Vision-Language Models
Stanford University（斯坦福大学）；University of Wisconsin–Madison（威斯康星大学麦迪逊分校）；Instituto Tecnológico de Monterrey（蒙特雷技术学院）；Monash University（墨尔本大学）；University of Cambridge（剑桥大学）；Shanghai Jiao Tong University（上海交通大学）；Shandong University（山东大学）
Multi-View Synergistic Learning with Vision-Language Adaption for Low-Resource Biomedical Image Classification
Chongqing University of Technology（重庆理工大学）；Collge of Computer Science, Sichuan University（四川大学计算机学院）；Hebei University of Technology（河北工业大学）；Nanyang Technological University（南洋理工大学）；Centre for Frontier AI Research, Agency for Science, Technology and Research, Singapore（新加坡科技研究局前沿人工智能研究中心）
Surface Vision Transformers: Flexible Attention-Based Modelling of Biomedical Surfaces
King’s College London（伦敦国王学院）；Washington University Medical School（华盛顿大学医学院）；Imperial College London（帝国理工学院）；Technical University of Munich（慕尼黑工业大学）；University of Edinburgh（爱丁堡大学）
How Well Can General Vision-Language Models Learn Medicine By Watching Public Educational Videos?
Stanford University（斯坦福大学）；Together AI；University of California, Berkeley（加州大学伯克利分校）；Cedars-Sinai Medical Center（西德斯-西奈医学中心）；University of California, San Francisco（加州大学旧金山分校）
LLaDA-MedV: Exploring Large Language Diffusion Models for Biomedical Image Understanding
Arizona State University（亚利桑那州立大学）；Clemson University（克莱姆森大学）；LinkedIn Corporation（领英公司）；Washington University in St. Louis（圣路易斯华盛顿大学）；Morgan Stanley（摩根大通）
Parameter Efficient Fine-Tuning of Segment Anything Model for Biomedical Imaging
Georg-August-University Göttingen, Institute of Physics（哥廷根大学物理学院）；Georg-August-University Göttingen, Institute of Computer Science（哥廷根大学计算机科学学院）；CAIMed - Lower Saxony Center for AI & Causal Methods in Medicine, Göttingen（下萨克森人工智能与医学因果方法中心）；Cluster of Excellence Multiscale Bioimaging (MBExC), Georg-August-University Göttingen（多尺度生物成像集群（MBExC），哥廷根大学）
本文首次系统研究了面向生物医学图像分割的 SAM 参数高效微调方法，发现 PEFT 层的放置位置比层类型对效率影响更大，并提出了资源高效的微调方案。
Hertie Institute for AI in Brain Health, University of Tübingen（图宾根大学赫蒂人工智能脑健康研究所）；Tübingen AI Center（图宾根人工智能中心）；Institute for Applied Medical Informatics, University of Tübingen（图宾根大学应用医学信息学研究所）
Hyper-Convolution Networks for Biomedical Image Segmentation
Cornell University（康奈尔大学）；Cornell Tech（康奈尔理工学院）；Massachusetts Institute of Technology（麻省理工学院）；Massachusetts General Hospital（麻省总医院）；Harvard Medical School（哈佛医学院）
Does Bigger Mean Better? Comparitive Analysis of CNNs and Biomedical Vision Language Modles in Medical Diagnosis
University of Texas at Dallas（德克萨斯大学达拉斯分校）；Independent Researcher（独立研究者）；Duke University（杜克大学）；University of Michigan Ann Arbor（密歇根大学安娜堡分校）；Georgia Institute of Technology（佐治亚理工学院）；University of California, Irvine（加州大学 Irvine 分校）
Vision Transformer-Conditioned UNet for Domain-Adaptive Semantic Segmentation
Helmholtz Pioneer Campus, Helmholtz Munich, Neuherberg, Germany（海德堡先锋校园，海德堡穆恩奇，纽赫尔伯格，德国）；School of Computation, Information and Technology, TUM, Garching, Germany（计算、信息与技术学院，技术大学慕尼黑，冈辛，德国）；Department of Chemistry, TUM, Garching, Germany（化学系，技术大学慕尼黑，冈辛，德国）
本文提出 ViTC-UNet，通过可学习令牌和双向注意力解码器将预训练 ViT 表示条件化于 UNet，以提升生物医学语义分割的精度与适应性。
