Biomedical imaging modalities often produce high-resolution, multi-dimensional images that pose computational challenges for deep neural networks. These computational challenges are compounded when training transformers due to the self-attention operator, which scales quadratically with context length. Recent developments in long-context models have potential to alleviate these difficulties and enable more efficient application of transformers to large biomedical images, although a systematic evaluation on this topic is lacking. In this study, we investigate the impact of context length on biomedical image analysis and we evaluate the performance of recently proposed long-context models. We first curate a suite of biomedical imaging datasets, including 2D and 3D data for segmentation, denoising, and classification tasks. We then analyze the impact of context length on network performance using the Vision Transformer and Swin Transformer by varying patch size and attention window size. Our findings reveal a strong relationship between context length and performance, particularly for pixel-level prediction tasks. Finally, we show that recent long-context models demonstrate significant improvements in efficiency while maintaining comparable performance, though we highlight where gaps remain. This work underscores the potential and challenges of using long-context models in biomedical imaging.
From $\mathcal{O}(n^{2})$ to $\mathcal{O}(n)$ Parameters: Quantum Self-Attention in Vision Transformers for Biomedical Image Classification
UCL Hawkes Institute(UCL哈维斯研究所);Department of Medical Physics and Biomedical Engineering(医学物理与生物医学工程系);University College London(伦敦大学学院);University College London Hospitals NHS Foundation Trust(伦敦大学学院医院国家健康服务信托基金会);Human Physiology and Performance Laboratory (HPPL)(人体生理与表现实验室);Centre for Peri-operative Medicine(围手术期医学中心);Department of Targeted Intervention(精准干预系);Division of Surgery and Interventional Science(外科与介入科学 division)
Advancing Italian Biomedical Information Extraction with Transformers-based Models: Methodological Insights and Multicenter Practical Application
IRCCS Istituto Centro San Giovanni di Dio Fatebenefratelli(IRCCS 圣乔瓦尼迪奥法特贝内弗拉特利研究所);University of Pavia(帕维亚大学);IRCCS Istituto Auxologico Italiano(IRCCS 意大利奥索洛吉科研究所);Università degli Studi di Milano(米兰大学);IRCCS Istituti Clinici Scientifici Maugeri(IRCCS 莫盖里临床科学研究所);IRCCS Mondino Foundation(IRCCS 蒙迪诺基金会)
R-GAT: Cancer Document Classification Leveraging Graph-Based Residual Network for Scenarios with Limited Data
Department of Computer Science and Engineering, Mississippi State University(密苏里州立大学计算机科学与工程系);Department of Information Technology, Universiti Tenaga Nasional(国家技术大学信息科技系);School of Electrical and Mechanical Engineering, University of Portsmouth(普利茅斯大学电气与机械工程学院);Department of Computer Science, The University of Alabama(阿拉巴马大学计算机科学系)
Extracting Protein-Protein Interactions (PPIs) from Biomedical Literature using Attention-based Relational Context Information
Computational Science Initiative(计算科学计划);Brookhaven National Laboratory(布鲁克海文国家实验室);Joint Genome Institute(联合基因组研究所);Lawrence Berkeley National Laboratory(劳伦斯伯克利国家实验室)
Multi-Compound Transformer for Accurate Biomedical Image Segmentation
The University of Hong Kong(香港大学);Shenzhen Research Institute of Big Data, The Chinese University of Hong Kong (Shenzhen)(香港中文大学(深圳)深圳大数据研究院);SenseTime Research(商汤科技研究院)
Surface Vision Transformers: Flexible Attention-Based Modelling of Biomedical Surfaces
King’s College London(伦敦国王学院);Washington University Medical School(华盛顿大学医学院);Imperial College London(帝国理工学院);Technical University of Munich(慕尼黑工业大学);University of Edinburgh(爱丁堡大学)
TTT-Unet: Enhancing U-Net with Test-Time Training Layers for Biomedical Image Segmentation
Lehigh University(里海大学);Northeastern University(东北大学);University of Notre Dame(圣母大学);University of Georgia(佐治亚大学);Massachusetts General Hospital(麻省总医院);Harvard Medical School(哈佛医学院)
Vision Transformer-Conditioned UNet for Domain-Adaptive Semantic Segmentation
Helmholtz Pioneer Campus, Helmholtz Munich, Neuherberg, Germany(海德堡先锋校园,海德堡穆恩奇,纽赫尔伯格,德国);School of Computation, Information and Technology, TUM, Garching, Germany(计算、信息与技术学院,技术大学慕尼黑,冈辛,德国);Department of Chemistry, TUM, Garching, Germany(化学系,技术大学慕尼黑,冈辛,德国)
本文提出 ViTC-UNet,通过可学习令牌和双向注意力解码器将预训练 ViT 表示条件化于 UNet,以提升生物医学语义分割的精度与适应性。
U-Mamba: Enhancing Long-range Dependency for Biomedical Image Segmentation
Peter Munk Cardiac Centre, University Health Network(彼得·蒙克心脏中心,大学健康网络);University of Toronto(多伦多大学);Vector Institute for Artificial Intelligence(向量人工智能研究所)
SSVT: Self-Supervised Vision Transformer For Eye Disease Diagnosis Based On Fundus Images
Beihang University(北京航空航天大学);Beijing Tongren Hospital, Capital Medical University(首都医科大学附属北京同仁医院);The University of Melbourne(墨尔本大学);University of Cambridge(剑桥大学);Imperial College London(帝国理工学院)
Variable-Length Tokenization via Learnable Global Merging for Diffusion Transformers
Kim Jaechul Graduate School of AI, KAIST, Daejeon, South Korea(韩国科学技术院金载哲人工智能研究生院,大田,韩国);School of Computing, KAIST, Daejeon, South Korea(韩国科学技术院计算学院,大田,韩国)
Going Beyond U-Net: Assessing Vision Transformers for Semantic Segmentation in Microscopy Image Analysis
Institute of Computer Science, University of Tartu(塔尔图大学计算机科学研究所);Center for Artificial Intelligence, Technical University of Applied Sciences Würzburg-Schweinfurt(维尔茨堡-施韦因富特应用技术大学人工智能中心);STACC
Parameter Efficient Fine-Tuning of Segment Anything Model for Biomedical Imaging
Georg-August-University Göttingen, Institute of Physics(哥廷根大学物理学院);Georg-August-University Göttingen, Institute of Computer Science(哥廷根大学计算机科学学院);CAIMed - Lower Saxony Center for AI & Causal Methods in Medicine, Göttingen(下萨克森人工智能与医学因果方法中心);Cluster of Excellence Multiscale Bioimaging (MBExC), Georg-August-University Göttingen(多尺度生物成像集群(MBExC),哥廷根大学)
本文首次系统研究了面向生物医学图像分割的 SAM 参数高效微调方法,发现 PEFT 层的放置位置比层类型对效率影响更大,并提出了资源高效的微调方案。
Predicting Genetic Mutation from Whole Slide Images via Biomedical-Linguistic Knowledge Enhanced Multi-label Classification
University of British Columbia(不列颠哥伦比亚大学);Sun Yat-sen University Cancer Center(中山大学肿瘤防治中心);Stanford University(斯坦福大学);Australian Artificial Intelligence Institute, University of Technology Sydney(悉尼科技大学澳大利亚人工智能研究所);School of Intelligent Systems Engineering, Sun Yat-sen University(中山大学智能工程学院);School of Computer Science and Engineering, Sun Yat-sen University(中山大学计算机科学与工程学院);Mohamed bin Zayed University of Artificial Intelligence (MBZUAI)(穆罕默德·本·扎耶德人工智能大学)
LLaDA-MedV: Exploring Large Language Diffusion Models for Biomedical Image Understanding
Arizona State University(亚利桑那州立大学);Clemson University(克莱姆森大学);LinkedIn Corporation(领英公司);Washington University in St. Louis(圣路易斯华盛顿大学);Morgan Stanley(摩根大通)
xLSTM-UNet can be an Effective 2D & 3D Medical Image Segmentation Backbone with Vision-LSTM (ViL) better than its Mamba Counterpart
College of Computer Science and Technology, Zhejiang University(浙江大学计算机科学与技术学院);KOKONI, Moxin (Huzhou) Tech. Co., LTD.(墨芯(湖州)科技有限公司KOKONI);School of Information Engineering, Huzhou University(湖州师范学院信息工程学院);Information Systems Technology and Design Pillar, Singapore University of Technology and Design(新加坡科技设计大学信息系统技术与设计 pillar);School of Information Science and Technology, University of Science and Technology of China(中国科学技术大学信息科学技术学院);School of Software Technology, Zhejiang University(浙江大学软件学院)