通过有限计算资源下的动量自蒸馏提升医学视觉-语言预训练
Boosting Medical Vision-Language Pretraining via Momentum Self-Distillation under Limited Computing Resources
AI总结：
本研究通过动量自蒸馏与梯度累积提升医学视觉-语言预训练效率，实现高效多模态学习并提升零样本分类和少量样本适应性能。
摘要：
在医疗健康领域，获取详细标注具有挑战性，凸显了需要鲁棒的视觉-语言模型（VLMs）的重要性。预训练的VLMs能够对小数据集进行微调或进行零样本推理，实现性能与任务特定模型相当。对比学习（CL）是训练VLMs的关键范式，但本质上需要大批次大小才能有效学习，使其计算需求高，通常局限于资源充足的机构。此外，由于医疗领域数据有限，训练过程中需要优先从数据和模型中提取知识以提高性能。因此，我们专注于利用动量方法结合蒸馏，同时解决计算效率和知识提取问题。我们的贡献可以总结如下：（1）利用动量自蒸馏增强多模态学习；（2）将动量机制与梯度累积结合，扩大有效批次大小而不增加资源消耗。我们的方法在零样本分类中与最先进（SOTA）方法具有竞争力，同时在少量样本适应中实现超过90%的AUC-ROC，并在检索任务中提升2-3%。重要的是，我们的方法在单个GPU上实现高训练效率，同时保持合理的训练时间。我们的方法旨在通过减少资源需求来推进高效的多模态学习，同时在SOTA方法上提高性能。我们的方法实现可在https://github.com/phphuc612/MSD上找到。
英文摘要：
In medical healthcare, obtaining detailed annotations is challenging, highlighting the need for robust Vision-Language Models (VLMs). Pretrained VLMs enable fine-tuning on small datasets or zero-shot inference, achieving performance comparable to task-specific models. Contrastive learning (CL) is a key paradigm for training VLMs but inherently requires large batch sizes for effective learning, making it computationally demanding and often limited to well-resourced institutions. Moreover, with limited data in healthcare, it is important to prioritize knowledge extraction from both data and models during training to improve performance. Therefore, we focus on leveraging the momentum method combined with distillation to simultaneously address computational efficiency and knowledge exploitation. Our contributions can be summarized as follows: (1) leveraging momentum self-distillation to enhance multimodal learning, and (2) integrating momentum mechanisms with gradient accumulation to enlarge the effective batch size without increasing resource consumption. Our method attains competitive performance with state-of-the-art (SOTA) approaches in zero-shot classification, while providing a substantial boost in the few-shot adaption, achieving over 90% AUC-ROC and improving retrieval tasks by 2-3%. Importantly, our method achieves high training efficiency with a single GPU while maintaining reasonable training time. Our approach aims to advance efficient multimodal learning by reducing resource requirements while improving performance over SOTA methods. The implementation of our method is available at https://github.com/phphuc612/MSD .
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
