arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2510.21175cs.AI
收藏

无记忆持续学习:基于零空间适配的零样本视觉-语言模型

Memory-Free Continual Learning with Null Space Adaptation for Zero-Shot Vision-Language Models

Yujin Jo, Taesup Kim

AI总结:

针对零样本视觉-语言模型在动态环境中的灾难性遗忘问题,提出无记忆持续学习框架NuSA-CL,通过低秩适配与零空间约束保留原有知识,兼顾零样本能力与持续学习性能。

摘要:

预训练的视觉-语言模型(VLMs),如CLIP,已经展现出卓越的零样本泛化能力,使得它们无需额外的任务特定训练即可部署到广泛的现实世界任务中。然而,在环境不断演变或出现新类别的真实部署场景中,这些模型不可避免地会面临分布偏移和新任务。在这种情况下,静态的零样本能力是不够的,越来越需要持续学习方法,使模型能够随时间适应,同时避免灾难性遗忘。我们提出了NuSA-CL(用于持续学习的零空间适配),这是一个轻量级、无记忆的持续学习框架,旨在解决这一挑战。NuSA-CL采用低秩适配,并将任务特定的权重更新约束在模型当前参数的近似零空间内。这一策略最小化了对先前获取知识的干扰,有效保留了原始模型的零样本能力。与依赖重放缓冲区或昂贵蒸馏的方法不同,NuSA-CL施加了最小的计算和内存开销,使其在资源受限的现实世界持续学习环境中部署变得切实可行。实验表明,我们的框架不仅有效保留了零样本迁移能力,而且在持续学习基准上取得了极具竞争力的性能。这些结果使NuSA-CL成为在现实应用中持续演进的零样本VLM的一种实用且可扩展的解决方案。

英文摘要:

Pre-trained vision-language models (VLMs), such as CLIP, have demonstrated remarkable zero-shot generalization, enabling deployment in a wide range of real-world tasks without additional task-specific training. However, in real deployment scenarios with evolving environments or emerging classes, these models inevitably face distributional shifts and novel tasks. In such contexts, static zero-shot capabilities are insufficient, and there is a growing need for continual learning methods that allow models to adapt over time while avoiding catastrophic forgetting. We introduce NuSA-CL (Null Space Adaptation for Continual Learning), a lightweight memory-free continual learning framework designed to address this challenge. NuSA-CL employs low-rank adaptation and constrains task-specific weight updates to lie within an approximate null space of the model's current parameters. This strategy minimizes interference with previously acquired knowledge, effectively preserving the zero-shot capabilities of the original model. Unlike methods relying on replay buffers or costly distillation, NuSA-CL imposes minimal computational and memory overhead, making it practical for deployment in resource-constrained, real-world continual learning environments. Experiments show that our framework not only effectively preserves zero-shot transfer capabilities but also achieves highly competitive performance on continual learning benchmarks. These results position NuSA-CL as a practical and scalable solution for continually evolving zero-shot VLMs in real-world applications.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑