视觉语言模型的黑盒持续学习
Black-Box Continual Learning for Vision-Language Models
AI总结：
提出黑盒持续学习基准Black-CL，仅通过输出嵌入或logits进行学习，并设计BETA方法，通过优化文本原型实现与白盒方法相当的性能。
摘要：
视觉语言模型（VLM）在动态环境中的快速部署要求其具备持续学习而不遗忘的能力。然而，传统的持续学习（CL）设置通常依赖白盒范式，这一范式因向云端托管模型的转变而日益失效。本文提出Black-CL，一个更贴近现实的VLM基准，它强制执行三个主要的现实挑战：权重和架构不可访问、计算受限以及任务无关推理。学习者只能查询输出嵌入或logits，无法通过骨干网络进行梯度流动或结构修改。当前的CL方法依赖于骨干反向传播或复杂的参数扩展，与这些约束根本不相容。在此设置下，我们提出BETA，一个简单而有效的基线，其关键洞察在于仅优化文本原型即可应对CL的复杂性。BETA集成三个核心组件：语义投影累积（SPA）用于增量知识获取、潜在分布重放（LDR）用于锚定嵌入空间以防止灾难性遗忘，以及测试时原型适应（TTPA）用于动态的实例感知边界细化。在十个不同数据集和多种骨干网络上的大量实验表明，BETA显著优于现有的黑盒调优方法。值得注意的是，仅用0.05M可训练参数（相比竞争方法减少180--3000倍），BETA达到了与白盒CL方法相当甚至更优的性能。我们相信Black-CL和BETA为持续学习的未来进展提供了基础框架，并加速了持续学习从学术界向现实系统的过渡。
英文摘要：
The rapid deployment of Vision-Language Models (VLMs) in dynamic environments necessitates the ability to learn continuously without forgetting. However, traditional continual learning (CL) settings often rely on white-box paradigms, which is increasingly invalidated by the shift toward cloud-hosted models. In this paper, we introduce Black-CL, a more realistic benchmark for VLMs that enforces three primary real-world challenges: weight and architecture inaccessibility, constrained computation, and task-agnostic inference. The learner can query only output embeddings or logits, with no gradient flow through or structural modification of the backbone. Current CL methodologies, which rely on backbone backpropagation or complex parameter expansion, are fundamentally incompatible with these constraints. Under this setting, we propose BETA, a simple yet effective baseline built on the key insight that solely optimizing textual prototypes can navigate the complexities of CL. BETA integrates three core components: Semantic Projection Accumulation (SPA) for incremental knowledge acquisition, Latent Distribution Replay (LDR) for anchoring the embedding space against catastrophic forgetting, and Test-Time Prototype Adaptation (TTPA) for dynamic, instance-aware boundary refinement. Extensive experiments across ten diverse datasets and various backbones demonstrate that BETA significantly outperforms existing black-box tuners. Remarkably, with only 0.05 M trainable parameters, a 180--3000$\times$ reduction compared to competitive methods, BETA achieves performance on par with or even exceeding white-box CL methods. We believe Black-CL and BETA provide a foundational framework for future advancements in continual learning and accelerates the transition of continual learning from academia to real-world systems.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
