分支还是层？零阶优化用于视觉-语言模型的持续学习
Branch, or Layer? Zeroth-Order Optimization for Continual Learning of Vision-Language Models
AI总结：
本文提出一种基于零阶优化的视觉-语言模型持续学习方法，通过模态感知策略提升模型逃避局部极小值的能力，实验表明在四个基准测试中取得最佳性能。
摘要：
视觉-语言持续学习（VLCL）因其强大的鲁棒性而受到广泛关注，参数高效微调（PEFT）策略的采用使这些模型能够以显著减少的资源消耗实现竞争性的性能。然而，主导的第一阶（FO）优化容易将模型困在次优局部极小值中，尤其是在PEFT内的有限探索子空间中。为克服这一挑战，本文首次系统地探索了在PEFT-based VLCL中采用零阶（ZO）优化。我们首先识别了在VLCL中采用朴素全零阶优化的不兼容性，原因在于优化过程的不稳定性。接着，我们研究了从模态分支到细粒度层的零阶优化应用，以确定最佳策略。此外，一个关键的理论洞察揭示了在VLCL的零阶优化过程中，视觉模态的方差高于语言模态，我们提出了一种模态感知的零阶策略，该策略在零阶优化中采用梯度符号归一化，并限制视觉模态的扰动以进一步提高性能。得益于零阶优化的采用，基于PEFT的VLCL在优化过程中具有更好的逃避局部极小值的能力，广泛的实验表明，我们的方法在四个基准测试中取得了最先进的结果。
英文摘要：
Vision-Language Continual Learning (VLCL) has attracted significant research attention for its robust capabilities, and the adoption of Parameter-Efficient Fine-Tuning (PEFT) strategies is enabling these models to achieve competitive performance with substantially reduced resource consumption. However, dominated First-Order (FO) optimization is prone to trap models in suboptimal local minima, especially in limited exploration subspace within PEFT. To overcome this challenge, this paper pioneers a systematic exploration of adopting Zeroth-Order (ZO) optimization for PEFT-based VLCL. We first identify the incompatibility of naive full-ZO adoption in VLCL due to optimization process instability. We then investigate the application of ZO optimization from a modality branch-wise to a fine-grained layer-wise across various training units to identify an optimal strategy. Besides, a key theoretical insight reveals that vision modality exhibit higher variance than language counterparts in VLCL during the ZO optimization process, and we propose a modality-aware ZO strategy, which adopts gradient sign normalization in ZO and constrains vision modality perturbation to further improve performance. Benefiting from the adoption of ZO optimization, PEFT-based VLCL fulfills better ability to escape local minima during the optimization process, extensive experiments on four benchmarks demonstrate that our method achieves state-of-the-art results.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
