动态跨模态提示生成用于多模态持续指令微调
Dynamic Cross-Modal Prompt Generation for Multimodal Continual Instruction Tuning
AI总结：
本文提出DRAPE框架，通过生成连续实例特定的软提示提升多模态持续指令微调性能，采用跨模态注意力和投影梯度投影减少遗忘，实验显示优于现有基线方法。
摘要：
多模态大语言模型（MLLMs）通过指令微调实现强性能，但现实部署需在连续任务中扩展能力。本文提出DRAPE（动态跨模态提示生成），通过生成连续实例特定的软提示提升多模态持续指令微调性能。DRAPE从文本指令和视觉补丁特征进行跨模态注意力，生成查询-图像条件提示，预置于冻结的LLM前。为减少连续更新中的遗忘，DRAPE对共享投影器应用null-space梯度投影，并使用CLIP基的原型路由进行任务标签无关的生成器选择。实验表明DRAPE在MCIT基准上达到现有提示基和LoRA基连续学习基线的最先进性能。
英文摘要：
Multimodal Large Language Models (MLLMs) achieve strong performance through instruction tuning, yet real-world deployment often requires continual capability expansion across sequential tasks. In such scenarios, Multimodal Continual Instruction Tuning (MCIT) aims to acquire new capabilities while limiting catastrophic forgetting. Existing methods mainly follow a module-composition paradigm: they maintain task-level prompts or LoRA experts and dynamically route or aggregate a subset of them at inference. However, samples within the same task can still differ substantially in visual scenes, question intents, and reasoning demands. This motivates instance-level adaptation to individual query-image pairs rather than only selecting or combining task-level modules. To this end, we propose DRAPE (Dynamic Cross-Modal Prompt Generation), a prompt-learning framework that synthesizes continuous instance-specific soft prompts for MCIT. Instead of selecting prompts from a fixed pool, DRAPE derives prompt queries from the textual instruction and cross-attends to visual patch features, producing query-image conditioned prompts that are prepended to the frozen LLM. To mitigate forgetting during sequential updates, DRAPE applies null-space gradient projection to the shared projector and uses CLIP-based prototype routing for task-label-free generator selection at inference. Extensive experiments on MCIT benchmarks show that DRAPE achieves state-of-the-art performance among representative prompt-based and LoRA-based continual-learning baselines.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
