EvoPrompt: 引导提示演化以适应视觉-语言模型
Guided Prompt Evolution for Vision-Language Models Adaptation
AI总结：
提出EvoPrompt框架，通过引导提示演化路径并解耦低秩更新为方向和幅度分量，实现视觉-语言模型在少样本学习中的遗忘-free微调，同时保持零样本能力。
摘要：
大规模视觉-语言模型（VLM）在有限标注数据下适应下游任务仍然是一个重大挑战。虽然参数高效的提示学习方法提供了一条有希望的路径，但它们常常遭受预训练知识的灾难性遗忘。为了解决这一限制，我们的工作基于一个洞察：控制提示的演化路径对于遗忘-free适应至关重要。为此，我们提出了EvoPrompt，一个旨在明确引导提示轨迹以进行知识保留微调的新型框架。具体来说，我们的方法采用模态共享提示投影器（MPP）从统一嵌入空间生成层次化提示。关键的是，一种演化训练策略将低秩更新解耦为方向和幅度分量，保留早期学习的语义方向而仅调整其幅度，从而使提示能够在不丢弃基础知识的情况下演化。这一过程通过特征几何正则化（FGR）进一步稳定，该正则化强制特征去相关以防止表示崩溃。大量实验表明，EvoPrompt在少样本学习中实现了最先进的性能，同时稳健地保留了预训练VLM的原始零样本能力。
英文摘要：
The adaptation of large-scale vision-language models (VLMs) to downstream tasks with limited labeled data remains a significant challenge. While parameter-efficient prompt learning methods offer a promising path, they often suffer from catastrophic forgetting of pre-trained knowledge. Toward addressing this limitation, our work is grounded in the insight that governing the evolutionary path of prompts is essential for forgetting-free adaptation. To this end, we propose EvoPrompt, a novel framework designed to explicitly steer the prompt trajectory for knowledge-preserving fine-tuning. Specifically, our approach employs a Modality-Shared Prompt Projector (MPP) to generate hierarchical prompts from a unified embedding space. Critically, an evolutionary training strategy decouples low-rank updates into directional and magnitude components, preserving early-learned semantic directions while only adapting their magnitude, thus enabling prompts to evolve without discarding foundational knowledge. Extensive experiments demonstrate that EvoPrompt achieves state-of-the-art performance in few-shot learning while robustly preserving the original zero-shot capabilities of pre-trained VLMs.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
