arXiv 2508.116732025-08-08cs.LGcs.AIcs.CVcs.MM

基于LoRA的对比正则化用于多模态生物医学图像增量学习

Contrastive Regularization over LoRA for Multimodal Biomedical Image Incremental Learning

School of Computer Science and Engineering, Central South University（计算机科学与工程学院，中南大学）；School of Automation, Central South University（自动化学院，中南大学）；School of Computer Science and Technology, Beijing Jiaotong University（计算机科学与技术学院，北京交通大学）

Haojie Zhang, Yixiong Liang, Hulin Kuang, Lihui Cen, Zhe Qu, Yigang Cen, Min Zeng, Shichao Kan

备注：: 10 pages, 3 figures, submitted to ACM Multimedia 2025
DOI：: 10.1145/3746027.3754917

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

本文提出MSLoRA-CR方法，通过模态特定LoRA微调与对比正则化，在保持预训练大型视觉语言模型冻结的前提下，解决多模态生物医学图像增量学习中的知识保留与跨模态迁移问题，性能较现有方法提升1.88%。

摘要：

多模态生物医学图像增量学习（MBIIL）对于处理生物医学领域中的多样化任务和模态至关重要，因为为每种模态或任务分别训练独立模型会显著增加推理成本。现有的增量学习方法主要关注单一模态内的任务扩展，而MBIIL旨在跨模态增量地训练一个统一模型。MBIIL面临两个挑战：I）如何在增量更新过程中保留先前学到的知识？II）如何有效利用从现有模态中获得的知识来支持新模态？为了解决这些挑战，我们提出了MSLoRA-CR，该方法微调模态特定的LoRA模块，同时引入对比正则化以增强模态内知识共享并促进模态间知识区分。我们的方法建立在一个大型视觉语言模型（LVLM）之上，保持预训练模型冻结，同时为每种模态或任务增量地适配新的LoRA模块。在生物医学图像增量学习上的实验表明，MSLoRA-CR优于为每种模态分别训练模型的最先进（SOTA）方法以及通用增量学习方法（增量微调LoRA）。具体而言，与无约束的增量学习方法相比，MSLoRA-CR在整体性能上实现了1.88%的提升，同时保持了计算效率。我们的代码公开在https://github.com/VentusAislant/MSLoRA_CR。

英文摘要：

Multimodal Biomedical Image Incremental Learning (MBIIL) is essential for handling diverse tasks and modalities in the biomedical domain, as training separate models for each modality or task significantly increases inference costs. Existing incremental learning methods focus on task expansion within a single modality, whereas MBIIL seeks to train a unified model incrementally across modalities. The MBIIL faces two challenges: I) How to preserve previously learned knowledge during incremental updates? II) How to effectively leverage knowledge acquired from existing modalities to support new modalities? To address these challenges, we propose MSLoRA-CR, a method that fine-tunes Modality-Specific LoRA modules while incorporating Contrastive Regularization to enhance intra-modality knowledge sharing and promote inter-modality knowledge differentiation. Our approach builds upon a large vision-language model (LVLM), keeping the pretrained model frozen while incrementally adapting new LoRA modules for each modality or task. Experiments on the incremental learning of biomedical images demonstrate that MSLoRA-CR outperforms both the state-of-the-art (SOTA) approach of training separate models for each modality and the general incremental learning method (incrementally fine-tuning LoRA). Specifically, MSLoRA-CR achieves a 1.88% improvement in overall performance compared to unconstrained incremental learning methods while maintaining computational efficiency. Our code is publicly available at https://github.com/VentusAislant/MSLoRA_CR.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。