ProtoAda: 原型引导的自适应适配器扩展与几何整合用于多模态持续指令微调
ProtoAda: Prototype-Guided Adaptive Adapter Expansion and Geometric Consolidation for Multimodal Continual Instruction Tuning
AI总结：
提出ProtoAda框架，通过格式感知任务原型和几何感知参数整合，解决多模态持续指令微调中任务路由错误和梯度干扰问题。
摘要：
多模态大语言模型通过指令微调取得了强大性能，但实际部署需要它们持续获取新的视觉语言能力，这使得多模态持续指令微调至关重要。为了减少任务间干扰并促进协作，近期方法常采用稀疏架构，如基于图像-文本相似度路由的LoRA专家混合。然而，具有不同响应结构的任务可能共享高度相似的视觉语言语义，从而被错误地路由到同一专家；仅凭图像-文本相似度不足以进行可靠的任务分配。例如，一个需要坐标预测的定位任务专家，在学习语义相似的VQA任务后，可能偏向于生成短文本答案。这种格式盲目的任务分配将异构响应类型整合到共享参数中，引发梯度干扰和无效的专家协作。为解决此问题，我们提出ProtoAda，一种原型引导的自适应微调框架。ProtoAda引入格式感知任务原型，使任务分配和路由与任务语义及输出结构对齐，并以几何感知方式整合格式兼容的更新，有效重用并逐步优化现有参数。在多个基准上的大量实验表明，ProtoAda取得了优越性能，尤其是在答案结构易被顺序微调破坏的任务上。
英文摘要：
Multimodal Large Language Models (MLLMs) achieve strong performance through instruction tuning, but real-world deployment requires them to continually acquire new vision-language capabilities, making Multimodal Continual Instruction Tuning (MCIT) essential. To reduce inter-task interference and promote collaboration, recent methods often employ sparse architectures like Mixture of LoRA Experts with image-text similarity routing. However, tasks with distinct response structures could share highly similar visual-linguistic semantics and thus be wrongly routed to the same expert; image-text similarity alone is insufficient for reliable task assignment. For example, an expert in a grounding task requiring coordinate prediction may be biased toward producing short textual answers after learning semantically similar VQA tasks. This format-blind task assignment integrates heterogeneous response types into shared parameters, inducing gradient interference and ineffective expert collaboration. To address this problem, we propose ProtoAda, a prototype-guided adaptive tuning framework. ProtoAda introduces format-aware task prototypes to align task assignment and routing with both task semantics and output structure, and further consolidates format-compatible updates in a geometry-aware manner to effectively reuse and progressively refine existing parameters. Extensive experiments on multiple benchmarks demonstrate that ProtoAda achieves superior performance, especially on tasks whose answer structures are easily corrupted by sequential tuning.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
