面向生成式视觉-语言模型持续学习的指令锚定视觉投影器
Instruction-Grounded Visual Projectors for Continual Learning of Generative Vision-Language Models
AI总结：
针对生成式VLM持续学习中视觉投影器更新导致指令被忽视的问题，提出指令锚定的混合投影器框架，结合专家推荐与剪枝策略，提升模型指令遵循能力与持续学习效果。
摘要：
持续学习能让预训练生成式视觉-语言模型（VLM）在无需过往任务重训练数据的情况下，融入新任务的知识。近期方法通过更新视觉投影器，为新任务转换视觉信息，以此连接预训练视觉编码器与大语言模型。但这类调整可能导致模型更重视视觉输入而非语言指令，尤其是在学习文本指令类型重复的任务时。为解决语言指令被忽视的问题，我们提出一种新框架，将视觉信息的转换锚定在供语言模型使用的指令上。我们引入混合视觉投影器，每个投影器都作为基于给定指令上下文的专用视觉-语言转换专家，以适配新任务。为避免将专家用于不相关的指令上下文，我们提出专家推荐策略，可为与已学任务相似的任务复用专家。此外，我们引入专家剪枝，以缓解过往任务中累积激活的专家带来的干扰。在多种视觉-语言任务上的大量实验表明，我们的方法能生成遵循指令的响应，表现优于现有持续学习方法。
英文摘要：
Continual learning enables pre-trained generative vision-language models (VLMs) to incorporate knowledge from new tasks without retraining data from previous ones. Recent methods update a visual projector to translate visual information for new tasks, connecting pre-trained vision encoders with large language models. However, such adjustments may cause the models to prioritize visual inputs over language instructions, particularly learning tasks with repetitive types of textual instructions. To address the neglect of language instructions, we propose a novel framework that grounds the translation of visual information on instructions for language models. We introduce a mixture of visual projectors, each serving as a specialized visual-to-language translation expert based on the given instruction context to adapt to new tasks. To avoid using experts for irrelevant instruction contexts, we propose an expert recommendation strategy that reuses experts for tasks similar to those previously learned. Additionally, we introduce expert pruning to alleviate interference from the use of experts that cumulatively activated in previous tasks. Extensive experiments on diverse vision-language tasks demonstrate that our method outperforms existing continual learning approaches by generating instruction-following responses.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
