BOFA: 基于CLIP的类增量学习中的桥接层正交低秩融合
BOFA: Bridge-Layer Orthogonal Low-Rank Fusion for CLIP-Based Class-Incremental Learning
AI总结：
提出BOFA框架，通过将适应限制在CLIP的跨模态桥接层并使用正交低秩融合防止遗忘，无需额外参数或数据回放，实现高效类增量学习。
摘要：
类增量学习（CIL）旨在持续学习新类别而不遗忘先前获得的知识。视觉-语言模型如CLIP通过多模态监督提供强大的可迁移表示，使其在CIL中具有前景。然而，将CLIP应用于CIL面临两大挑战：（1）适应下游任务通常需要额外的可学习模块，增加模型复杂性和遗忘风险；（2）多模态表示具有互补优势，但现有方法尚未充分利用视觉和文本模态的有效集成。为解决这些问题，我们提出BOFA（桥接层正交融合适应），一种用于CIL的新框架。BOFA将所有模型适应限制在CLIP现有的跨模态桥接层，从而不增加额外参数或推理成本。为防止该层内的遗忘，它利用正交低秩融合，一种将参数更新约束在数学构造的与过去任务特征正交的低秩“安全子空间”的机制。这确保了无需数据回放即可稳定积累知识。此外，BOFA采用跨模态混合原型，将稳定的文本原型与从稳定适应的桥接层派生的视觉原型协同，提升分类性能。在标准基准上的大量实验表明，BOFA相比现有方法实现了卓越的准确性和效率。
英文摘要：
Class-Incremental Learning (CIL) aims to continually learn new categories without forgetting previously acquired knowledge. Vision-language models such as CLIP offer strong transferable representations via multi-modal supervision, making them promising for CIL. However, applying CLIP to CIL poses two major challenges: (1) adapting to downstream tasks often requires additional learnable modules, increasing model complexity and susceptibility to forgetting; and (2) while multi-modal representations offer complementary strengths, existing methods have yet to fully realize their potential in effectively integrating visual and textual modalities. To address these issues, we propose BOFA (Bridge-layer Orthogonal Fusion for Adaptation), a novel framework for CIL. BOFA confines all model adaptation exclusively to CLIP's existing cross-modal bridge-layer, thereby adding no extra parameters or inference cost. To prevent forgetting within this layer, it leverages Orthogonal Low-Rank Fusion, a mechanism that constrains parameter updates to a low-rank ``safe subspace" mathematically constructed to be orthogonal to past task features. This ensures stable knowledge accumulation without data replay. Furthermore, BOFA employs a cross-modal hybrid prototype that synergizes stable textual prototypes with visual counterparts derived from our stably adapted bridge-layer, enhancing classification performance. Extensive experiments on standard benchmarks show that BOFA achieves superior accuracy and efficiency compared to existing methods.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
