通过语义-几何保持实现视觉-语言模型的持续学习
Continual Learning with Vision-Language Models via Semantic-Geometry Preservation
AI总结：
本文提出SeGP-CL方法，通过构建对抗锚点和跨模态几何蒸馏，解决持续学习中的语义几何退化问题，提升模型稳定性和迁移能力。
摘要：
预训练视觉-语言模型（VLMs）的持续学习容易出现灾难性遗忘，现有方法在适应新任务时未能显式保持跨模态语义几何结构，导致新任务监督诱导几何失真。本文提出语义几何保持持续学习（SeGP-CL），通过构建对抗锚点和跨模态几何蒸馏，稳定文本参考框架，最终通过融合跨模态和视觉线索实现双路径推理。实验表明，SeGP-CL在五个持续学习基准上提升了稳定性和迁移能力，达到最先进的性能并更好地保持VLMs的语义几何结构。
英文摘要：
Continual learning of pretrained vision-language models (VLMs) is prone to catastrophic forgetting, yet current approaches adapt to new tasks without explicitly preserving the cross-modal semantic geometry inherited from pretraining and previous stages, allowing new-task supervision to induce geometric distortion. We observe that the most pronounced drift tends to concentrate in vulnerable neighborhoods near the old-new semantic interface, where shared visual patterns are easily re-explained by new textual semantics. To address this under an exemplar-free constraint, we propose Semantic Geometry Preservation for Continual Learning (SeGP-CL). SeGP-CL first probes the drift-prone region by constructing a compact set of adversarial anchors with dual-targeted projected gradient descent (DPGD), which drives selected new-task seeds toward old-class semantics while remaining faithful in raw visual space. During training, we preserve cross-modal structure by anchor-guided cross-modal geometry distillation (ACGD), and stabilize the textual reference frame across tasks via a lightweight text semantic-geometry regularization (TSGR). After training, we estimate anchor-induced raw-space drift to transfer old visual prototypes and perform dual-path inference by fusing cross-modal and visual cues. Extensive experiments on five continual learning benchmarks demonstrate that SeGP-CL consistently improves stability and forward transfer, achieving state-of-the-art performance while better preserving semantic geometry of VLMs. Code is available at: https://github.com/chiyuan-IVIPLab/SeGP-CL.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
