Boosting Memory Efficiency in Transfer Learning for High-Resolution Medical Image Classification
Department of Electronic and Electrical Engineering, Southern University of Science and Technology（电子与电气工程系，南方科技大学）；School of Biomedical Engineering, The University of British Columbia（生物医学工程学院，不列颠哥伦比亚大学）；Department of Electrical and Electronic Engineering, The University of Hong Kong（电气与电子工程系，香港大学）；Jiaxing Research Institute, Southern University of Science and Technology（嘉兴研究院，南方科技大学）
Yijin Huang, Pujin Cheng, Roger Tam, Xiaoying Tang
大规模预训练模型的成功使得微调成为在下游任务中实现显著提升的标准方法。然而，微调预训练模型的整个参数集成本高昂。参数高效迁移学习（PETL）最近作为一种经济高效的替代方案出现，用于将预训练模型适配到下游任务。尽管具有优势，但不断增长的模型规模和输入分辨率给PETL带来了挑战，因为训练内存消耗的降低效果不如参数使用量那样显著。在本文中，我们提出了Fine-grained Prompt Tuning plus (FPT+)，这是一种专为高分辨率医学图像分类设计的PETL方法，与其他PETL方法相比，它能显著减少训练内存消耗。FPT+通过训练一个轻量级侧网络，并通过细粒度提示和融合模块从大型预训练模型（LPM）中获取预训练知识来进行迁移学习。具体而言，我们冻结目标LPM并构建一个可学习的轻量级侧网络。冻结的LPM处理高分辨率图像以提取细粒度特征，而侧网络则采用相应的下采样低分辨率图像以最小化内存使用。为了使侧网络能够利用预训练知识，我们提出了细粒度提示和融合模块，它们协同工作以通过LPM的中间激活来汇总信息。我们在八个不同规模、模态和复杂度的医学图像数据集上评估了FPT+。实验结果表明，FPT+优于其他PETL方法，仅使用微调整个ViT-B模型所需可学习参数的1.03%和内存的3.18%。我们的代码可在https://github.com/YijinHuang/FPT获取。
英文摘要：
The success of large-scale pre-trained models has established fine-tuning as a standard method for achieving significant improvements in downstream tasks. However, fine-tuning the entire parameter set of a pre-trained model is costly. Parameter-efficient transfer learning (PETL) has recently emerged as a cost-effective alternative for adapting pre-trained models to downstream tasks. Despite its advantages, the increasing model size and input resolution present challenges for PETL, as the training memory consumption is not reduced as effectively as the parameter usage. In this paper, we introduce Fine-grained Prompt Tuning plus (FPT+), a PETL method designed for high-resolution medical image classification, which significantly reduces the training memory consumption compared to other PETL methods. FPT+ performs transfer learning by training a lightweight side network and accessing pre-trained knowledge from a large pre-trained model (LPM) through fine-grained prompts and fusion modules. Specifically, we freeze the LPM of interest and construct a learnable lightweight side network. The frozen LPM processes high-resolution images to extract fine-grained features, while the side network employs corresponding down-sampled low-resolution images to minimize the memory usage. To enable the side network to leverage pre-trained knowledge, we propose fine-grained prompts and fusion modules, which collaborate to summarize information through the LPM's intermediate activations. We evaluate FPT+ on eight medical image datasets of varying sizes, modalities, and complexities. Experimental results demonstrate that FPT+ outperforms other PETL methods, using only 1.03% of the learnable parameters and 3.18% of the memory required for fine-tuning an entire ViT-B model. Our code is available on https://github.com/YijinHuang/FPT.
National University of Singapore（新加坡国立大学）；National University Hospital（国立大学医院）；Zhejiang University（浙江大学）
In this paper, we propose a novel semi-supervised learning (SSL) framework named BoostMIS that combines adaptive pseudo labeling and informative active annotation to unleash the potential of medical image SSL models: (1) BoostMIS can adaptively leverage the cluster assumption and consistency regularization of the unlabeled data according to the current learning status. This strategy can adaptively generate one-hot "hard" labels converted from task model predictions for better task model training. (2) For the unselected unlabeled images with low confidence, we introduce an Active learning (AL) …
Boosting Few-Shot Learning with Disentangled Self-Supervised Learning and Meta-Learning for Medical Image Classification
Institute of Information Science and Technologies, National Research Council（国家研究委员会信息科学与技术研究所）；Department of Information Engineering, University of Pisa（比萨大学信息工程系）；School of Engineering, University of Edinburgh（爱丁堡大学工程学院）
Boosting Vision Semantic Density with Anatomy Normality Modeling for Medical Vision-language Pre-training
College of Computer Science and Technology, Zhejiang University（浙江大学计算机科学与技术学院）；DAMO Academy, Alibaba Group（阿里巴巴集团达摩院）；The First Affiliated Hospital of College of Medicine, Zhejiang University（浙江大学医学院附属第一医院）；Hupan Lab（虎盘实验室）
Boosting Medical Image-based Cancer Detection via Text-guided Supervision from Reports
DAMO Academy, Alibaba Group（阿里巴巴集团达摩学院）；Hefei Comprehensive National Science Center（合肥综合性国家科学中心）；Hupan Lab（湖畔实验室）；Sun Yat-sen University Cancer Center（中山大学肿瘤防治中心）；South China Hospital, Medical School, Shenzhen University（深圳大学医学院华南医院）
Fine-grained Prompt Tuning: A Parameter and Memory Efficient Transfer Learning Method for High-resolution Medical Image Classification
School of Biomedical Engineering, The University of British Columbia（不列颠哥伦比亚大学生物医学工程学院）；Department of Electrical and Electronic Engineering, The University of Hong Kong（香港大学电机电子工程学系）；Jiaxing Research Institute, Southern University of Science and Technology（南方科技大学嘉兴研究院）；Department of Electronic and Electrical Engineering, Southern University of Science and Technology（南方科技大学电子与电气工程系）
2.75D: Boosting learning by representing 3D Medical imaging to 2D features for small data
Netherlands Cancer Institute（荷兰癌症研究所）；Erasmus Medical Center Rotterdam（鹿特丹伊拉斯姆斯医学中心）；Radboud University Medical Center（拉德堡德大学医学中心）；Southern University of Science and Technology（南方科技大学）；Aberystwyth University（阿伯里斯特威斯大学）
Human visual attention has recently shown its distinct capability in boosting machine learning models. However, studies that aim to facilitate medical tasks with human visual attention are still scarce. To support the use of visual attention, this paper describes a novel deep learning model for visual saliency prediction on chest X-ray (CXR) images. To cope with data deficiency, we exploit the multi-task learning method and tackles disease classification on CXR simultaneously. For a more robust training process, we propose a further optimized multi-task learning scheme to better handle model o…
Hybrid Convolution Neural Network Integrated with Pseudo-Newton Boosting for Lumbar Spine Degeneration Detection
School of Computer Science and Engineering, Vellore Institute of Technology（韦洛尔理工大学计算机科学与工程学院）；Department of Information Science and Technology, College of Engineering Guindy, Anna University（安娜大学金迪工程学院信息科学与技术系）；Integrated Management Coastal Zones Research Institute, Universitat Politecnica de Valencia（瓦伦西亚理工大学海岸带综合管理研究所）
Australian Institute for Machine Learning（澳大利亚机器学习研究所）；University of Adelaide（阿德莱德大学）；Otto von Guericke University Magdeburg（马格德堡奥托·冯·格里克大学）
Real-world large-scale medical image analysis (MIA) datasets have three challenges: 1) they contain noisy-labelled samples that affect training convergence and generalisation, 2) they usually have an imbalanced distribution of samples per class, and 3) they normally comprise a multi-label problem, where samples can have multiple diagnoses. Current approaches are commonly trained to solve a subset of those problems, but we are unaware of methods that address the three problems simultaneously. In this paper, we propose a new training module called Non-Volatile Unbiased Memory (NVUM), which non-v…
Airdoc（鹰瞳科技）；Harbin Engineering University（哈尔滨工程大学）；IIAI（国际人工智能研究院）；Monash University（莫纳什大学）；The University of Sydney（悉尼大学）
Deep neural networks are known to be data-driven and label noise can have a marked impact on model performance. Recent studies have shown great robustness to classic image recognition even under a high noisy rate. In medical applications, learning from datasets with label noise is more challenging since medical imaging datasets tend to have asymmetric (class-dependent) noise and suffer from high observer variability. In this paper, we systematically discuss and define the two common types of label noise in medical images - disagreement label noise from inconsistency expert opinions and single-…
Convolutional Neural Networks (CNNs) usually use the same activation function, such as RELU, for all convolutional layers. There are performance limitations of just using RELU. In order to achieve better classification performance, reduce training and testing times, and reduce power consumption and memory usage, a new "Compressed Multi-function CNN" is developed. Google's Inception-V4, for example, is a very deep CNN that consists of 4 Inception-A blocks, 7 Inception-B blocks, and 3 Inception-C blocks. RELU is used for all convolutional layers. A new "Compressed Multi-function Inception-V4" (C…
Institute of Medical Informatics, University of Lübeck（吕贝克大学医学信息学研究所）
Currently generative adversarial networks (GANs) are rarely applied to medical images of large sizes, especially 3D volumes, due to their large computational demand. We propose a novel multi-scale patch-based GAN approach to generate large high resolution 2D and 3D images. Our key idea is to first learn a low-resolution version of the image and then generate patches of successively growing resolutions conditioned on previous scales. In a domain translation use-case scenario, 3D thorax CTs of size 512x512x512 and thorax X-rays of size 2048x2048 are generated and we show that, due to the constan…
Analog In-Memory Computing with Uncertainty Quantification for Efficient Edge-based Medical Imaging Segmentation
École nationale Supérieure d’Informatique（国立高等计算机学院）；IBM Research Europe（IBM欧洲研究院）；Yorktown High school（约克镇高中）；IBM T. J. Watson Research Center（IBM T. J. 沃森研究中心）
Unsupervised Anomaly Detection in Medical Images with a Memory-augmented Multi-level Cross-attentional Masked Autoencoder
Harvard University（哈佛大学）；Singapore Management University（新加坡管理大学）；Australian Institute for Machine Learning（澳大利亚机器学习研究所）；University of Adelaide（阿德莱德大学）；South Australian Health and Medical Research Institute（南澳大利亚健康与医学研究所）；University of Surrey（萨里大学）
LMU München（慕尼黑大学）；Technische Universität München（慕尼黑工业大学）；Johns Hopkins University（约翰斯·霍普金斯大学）
Fully Convolutional Neural Networks (F-CNNs) achieve state-of-the-art performance for image segmentation in medical imaging. Recently, squeeze and excitation (SE) modules and variations thereof have been introduced to recalibrate feature maps channel- and spatial-wise, which can boost performance while only minimally increasing model complexity. So far, the development of SE has focused on 2D images. In this paper, we propose `Project & Excite' (PE) modules that base upon the ideas of SE and extend them to operating on 3D volumetric images. `Project & Excite' does not perform global average po…
South China University of Technology（华南理工大学）；University of Utah（犹他大学）
We introduce the idea of inter-slice image augmentation whereby the numbers of the medical images and the corresponding segmentation labels are increased between two consecutive images in order to boost medical image segmentation accuracy. Unlike conventional data augmentation methods in medical imaging, which only increase the number of training samples directly by adding new virtual samples using simple parameterized transformations such as rotation, flipping, scaling, etc., we aim to augment data based on the relationship between two consecutive images, which increases not only the number b…
Accelerating Volumetric Medical Image Annotation via Short-Long Memory SAM 2
Department of Electrical and Computer Engineering, Duke University（电气与计算机工程系，杜克大学）；Department of Biostatistics and Bioinformatics, Duke University（生物统计学与生物信息学系，杜克大学）；Department of Biostatistics and Bioinformatics, Radiology, Electrical and Computer Engineering, and Computer Science, Duke University（生物统计学与生物信息学系，放射学，电气与计算机工程，和计算机科学，杜克大学）
本文提出 SLM-SAM 2 架构，通过分离的短期和长期记忆库及注意力模块解决 SAM 2 在体积医学图像标注中的错误传播问题，在四个数据集上显著提升了分割精度并减少了 60.575% 的掩码校正时间。
Knowledge Boosting: Rethinking Medical Contrastive Vision-Language Pre-Training
Southeast University（东南大学）；Nanjing University of Aeronautics and Astronautics（南京航空航天大学）；Case Western Reserve University（凯斯西储大学）；Centre de Recherche en Information Biomédicale Sino-Français（中法生物医学信息研究中心）
Boosting Semi-Supervised Medical Image Segmentation via Masked Image Consistency and Discrepancy Learning
School of Information and Communication Engineering, Beijing University of Posts and Telecommunications（北京邮电大学信息与通信工程学院）；School of Artificial Intelligence, Beijing University of Posts and Telecommunications（北京邮电大学人工智能学院）；School of Computer Science and Engineering, Southeast University（东南大学计算机科学与工程学院）
Boosting Medical Image Segmentation Performance with Adaptive Convolution Layer
LIPN-UMR-CNRS 7030（法国国家科学研究中心UMR 7030 LIPN实验室）；Sorbonne University Paris Nord（巴黎第十一大学（索邦大学巴黎北校区））；Amirkabir University of Technology（阿米尔卡比尔理工大学）；Université Paris-Est Créteil（巴黎东部克雷泰伊大学）
Boosting Ultrasound Image Classification via Attribute-Guided Dual-Branch Framework
School of Computer Science, National Engineering Research Center for Multimedia Software, Institute of Artificial Intelligence, Hubei Key Laboratory of Multimedia and Network Communication Engineering, Wuhan University（武汉大学计算机学院、国家多媒体软件工程技术研究中心、人工智能研究院、湖北省多媒体与网络通信工程重点实验室）
The scarcity of labeled data often impedes the application of deep learning to the segmentation of medical images. Semi-supervised learning seeks to overcome this limitation by exploiting unlabeled examples in the learning process. In this paper, we present a novel semi-supervised segmentation method that leverages mutual information (MI) on categorical distributions to achieve both global representation invariance and local smoothness. In this method, we maximize the MI for intermediate feature embeddings that are taken from both the encoder and decoder of a segmentation network. We first pro…
TM-UNet: Token-Memory Enhanced Sequential Modeling for Efficient Medical Image Segmentation
Dalian University of Technology（大连理工大学）；University of Lincoln（林肯大学）；University of Nottingham（诺丁汉大学）；Waseda University（早稻田大学）；University of Nottingham Ningbo China（宁波诺丁汉大学）；Yale University（耶鲁大学）
One-Shot Medical Video Object Segmentation via Temporal Contrastive Memory Networks
Wuhan University of Technology（武汉理工大学）；Shanghai Artificial Intelligence Laboratory（上海人工智能实验室）；MedAI Technology (Wuxi) Co. Ltd.（MedAI科技（无锡）有限公司）；Technical University of Munich（慕尼黑工业大学）
PPDM: Pixel Puzzling Diffusion Model for Speed and Memory Efficient Volumetric Medical Image Translation
Department of Radiology, Northwestern University（西北大学放射学系）；Department of Biomedical Engineering, Yale University（耶鲁大学生物医学工程系）；Department of Radiology and Biomedical Imaging, Yale School of Medicine（耶鲁医学院放射学与生物医学影像系）
Boosting Medical Vision-Language Pretraining via Momentum Self-Distillation under Limited Computing Resources
Faculty of Information Technology, University of Science, Ho Chi Minh City, Vietnam（信息科技学院，科学大学，胡志明市，越南）；Vietnam National University, Ho Chi Minh City, Vietnam（越南国家大学，胡志明市，越南）
SAMed-2: Selective Memory Enhanced Medical Segment Anything Model
Lehigh University, Bethlehem, PA, USA（莱斯大学）；Massachusetts General Hospital（麻省总医院）；Harvard Medical School（哈佛医学院）；University of Georgia, Athens, GA, USA（佐治亚大学）；University of Notre Dame, Notre Dame, IN, USA（Notre Dame 大学）
Similarity Memory Prior is All You Need for Medical Image Segmentation
School of Computer Science and Technology, Xinjiang University（新疆大学计算机科学与技术学院）；Xinjiang Multimodal Intelligent Processing and Information Security Engineering Technology Research Center（新疆多模态智能处理与信息安全工程技术研究中心）；Silk Road Multilingual Cognitive Computing International Cooperation Joint Laboratory, Xinjiang University（丝绸之路多语言认知计算国际合作联合实验室）
Conservatoire National des Arts et Métiers（法国国立工艺学院）；IRCAD（消化癌症研究所）
Transformer models achieve state-of-the-art results for image segmentation. However, achieving long-range attention, necessary to capture global context, with high-resolution 3D images is a fundamental challenge. This paper introduces the Full resolutIoN mEmory (FINE) transformer to overcome this issue. The core idea behind FINE is to learn memory tokens to indirectly model full range interactions while scaling well in both memory and computational costs. FINE introduces memory tokens at two levels: the first one allows full interaction between voxels within local image regions (patches), the …
University of Science and Technology of China（中国科学技术大学）；Damo Academy, Alibaba Group（阿里巴巴达摩院）；Rose-Hulman Institute of Technology（罗斯-霍曼理工学院）；Microsoft Research（微软研究院）；Institute for AI Industry Research (AIR), Tsinghua University（清华大学人工智能产业研究院）；State Key Laboratory of Networking and Switching Technology, Beijing University of Posts and Telecommunications（北京邮电大学网络与交换技术国家重点实验室）
Human brains are known to be capable of speeding up visual recognition of repeatedly presented objects through faster memory encoding and accessing procedures on activated neurons. For the first time, we borrow and distill such a capability into a semantic memory design, namely SMTM, to improve on-device CNN inference. SMTM employs a hierarchical memory architecture to leverage the long-tail distribution of objects of interest, and further incorporates several novel techniques to put it into effects: (1) it encodes high-dimensional feature maps into low-dimensional, semantic vectors for low-co…
Boosting Convolution with Efficient MLP-Permutation for Volumetric Medical Image Segmentation
Department of Computer Science and Engineering（计算机科学与工程系）；The Hong Kong University of Science and Technology（香港科学与技术大学）；HKUST Shenzhen-Hong Kong Collaborative Innovation Research Institute（HKUST深圳-香港协同创新研究院）
Memory-Supported Synergistic Adaptation for Training-Free Test-Time Medical Image Segmentation
School of Computer Science, University of Nottingham（诺丁汉大学计算机科学学院）；School of Computer Science and Information Engineering, Hefei University of Technology（合肥工业大学计算机科学与信息工程学院）；Information Systems Technology and Design Pillar, Singapore University of Technology and Design（新加坡科技设计大学信息系统技术与设计支柱）
Shifting Adaptation from Weight Space to Memory Space: A Memory-Augmented Agent for Medical Image Segmentation
Department of Electrical and Computer Engineering, University of California, Santa Barbara, Santa Barbara, CA 93106, USA（电子工程系，加州大学圣芭芭拉分校）；College of Engineering, Northeastern University, Boston, MA 02115, USA（工程学院，东北大学）；Department of Biomedical Engineering, New Jersey Institute of Technology, Newark, NJ 07102, USA（生物医学工程系，新泽西理工学院）；United Imaging Intelligence, Burlington, MA 01803, USA（联合影像智能公司）；Department of Radiation Oncology, Mayo Clinic, Scottsdale, AZ 85259, USA（放射肿瘤科，梅奥诊所）；Department of Radiology, Massachusetts General Hospital and Harvard Medical School, Boston, MA 02115, USA（放射科，麻省总医院和哈佛医学院）；School of Computing, University of Georgia, Athens, GA 30602, USA（计算学院，佐治亚大学）
Concept Alignment Contrast and Long-Short Prompt Memory for Test-Time Adaptation of SAM3 in Medical Image Segmentation
University of Electronic Science and Technology of China（电子科技大学）；Faculty of Information Technology, Monash University（莫纳什大学信息技术学院）；Shanghai Artificial Intelligence Laboratory（上海人工智能实验室）
