SEFE：面向多模态持续指令微调的表层与本质遗忘消除器
SEFE: Superficial and Essential Forgetting Eliminator for Multimodal Continual Instruction Tuning
AI总结：
本文针对多模态持续指令微调中的遗忘问题，区分表层遗忘与本质遗忘，提出ASD风格多样化范式与RegLoRA正则化方法，共同构成SEFE框架，实现最先进性能。
摘要：
多模态持续指令微调（MCIT）旨在使多模态大语言模型（MLLMs）能够增量学习新任务而不发生灾难性遗忘。本文对该情境下的遗忘进行了探索，将其划分为表层遗忘和本质遗忘。表层遗忘指模型的知识可能并未真正丢失，但由于后续任务答案风格的影响，其对先前任务的响应偏离了预期格式，导致结果不可用。相比之下，本质遗忘指模型给出了格式正确但事实不准确的答案，表明知识真正发生了丢失。评估本质遗忘需要首先解决表层遗忘，因为严重的表层遗忘会掩盖模型的知识状态。因此，我们首先引入答案风格多样化（ASD）范式，该范式定义了跨不同任务转换数据风格的标准化流程，将各任务的训练集统一为相似多样化的风格，以防止由风格偏移引起的表层遗忘。在此基础上，我们提出RegLoRA以缓解本质遗忘。RegLoRA通过施加正则化来稳定先前知识主要存储的关键参数，使模型能够保留已有能力。实验结果表明，我们的整体方法SEFE达到了最先进的性能。
英文摘要：
Multimodal Continual Instruction Tuning (MCIT) aims to enable Multimodal Large Language Models (MLLMs) to incrementally learn new tasks without catastrophic forgetting. In this paper, we explore forgetting in this context, categorizing it into superficial forgetting and essential forgetting. Superficial forgetting refers to cases where the model's knowledge may not be genuinely lost, but its responses to previous tasks deviate from expected formats due to the influence of subsequent tasks' answer styles, making the results unusable. By contrast, essential forgetting refers to situations where the model provides correctly formatted but factually inaccurate answers, indicating a true loss of knowledge. Assessing essential forgetting necessitates addressing superficial forgetting first, as severe superficial forgetting can obscure the model's knowledge state. Hence, we first introduce the Answer Style Diversification (ASD) paradigm, which defines a standardized process for transforming data styles across different tasks, unifying their training sets into similarly diversified styles to prevent superficial forgetting caused by style shifts. Building on this, we propose RegLoRA to mitigate essential forgetting. RegLoRA stabilizes key parameters where prior knowledge is primarily stored by applying regularization, enabling the model to retain existing competencies. Experimental results demonstrate that our overall method, SEFE, achieves state-of-the-art performance.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
