ModalPrompt：基于双模态引导提示的高效多模态持续指令调优
ModalPrompt: Towards Efficient Multimodal Continual Instruction Tuning with Dual-Modality Guided Prompt
AI总结：
针对大型多模态模型在动态新任务中持续学习时的知识遗忘与效率问题，提出双模态引导的提示学习框架ModalPrompt，通过提示融合迁移知识、提示选择管理复杂度，在MCIT基准上性能提升14.26%且推理速度快1.42倍。
摘要：
大型多模态模型（LMMs）通过学习混合指令数据集展现出卓越的多任务能力。然而，动态世界中会依次出现新任务，这促使LMMs具备多模态持续指令学习（MCIT）能力，尤其针对多样且具挑战性的生成任务。现有MCIT方法未充分利用LMMs的独特属性，且常以效率为代价换取性能。本文提出一种面向MCIT的新型提示学习框架，借助自然图文监督有效缓解对先前知识的遗忘，同时管理计算复杂度。具体而言，我们为每个任务学习提示，并利用双模态引导的高效提示融合实现知识迁移，通过提示选择管理复杂度。大量实验表明，我们的方法在MCIT基准上实现了14.26%的显著性能提升，且推理速度提升1.42倍，无需额外计算开销。代码可在https://github.com/AuroraZengfh/ModalPrompt获取。
英文摘要：
Large Multimodal Models (LMMs) exhibit remarkable multi-tasking ability by learning mixed instruction datasets. However, novel tasks would be encountered sequentially in dynamic world, which urges for equipping LMMs with multimodal continual instruction learning (MCIT) ability especially for diverse and challenging generative tasks. Existing MCIT methods do not fully exploit the unique attribute of LMMs and often gain performance at the expense of efficiency. In this paper, we propose a novel prompt learning framework for MCIT to effectively alleviate forgetting of previous knowledge while managing computational complexity with natural image-text supervision. Concretely, we learn prompts for each task and exploit efficient prompt fusion for knowledge transfer and prompt selection for complexity management with dual-modality guidance. Extensive experiments demonstrate that our approach achieves substantial +14.26% performance gain on MCIT benchmarks with remarkable $\times$ 1.42 inference speed free from growing computation. Code is available at https://github.com/AuroraZengfh/ModalPrompt.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
