LLaMA-Excitor：通过间接特征交互实现通用指令微调
LLaMA-Excitor: General Instruction Tuning via Indirect Feature Interaction
AI总结：
本文提出轻量级方法LLaMA-Excitor，通过自注意力旁路模块间接调整特征重要性，在不损害预训练能力的前提下实现通用指令微调，并在语言和多模态任务上取得显著提升。
摘要：
现有的LLM微调方法，如Adapter、Prefix-tuning和LoRA，通过引入额外模块或附加输入序列来注入新技能或知识，可能会损害LLM的固有能力。本文提出LLaMA-Excitor，一种轻量级方法，通过逐步对有价值信息给予更多关注来激发LLM更好地遵循指令的潜力。具体而言，LLaMA-Excitor在Transformer结构的自注意力计算过程中不直接改变中间隐藏状态。我们设计了Excitor块作为LLM自注意力中相似度分数计算的旁路模块，通过可学习提示重构键并改变值的重要性。LLaMA-Excitor确保对输入指令进行自适应分配额外注意力，从而在低质量指令遵循数据集上微调LLM时有效保留预训练知识。此外，我们统一了多模态微调和纯语言微调的建模，将LLaMA-Excitor扩展为强大的视觉指令跟随器，无需复杂的多模态对齐。我们提出的方法在纯语言和多模态微调实验场景中进行了评估。值得注意的是，LLaMA-Excitor是唯一在MMLU基准上实现显著提升（+6%）的同时保持基本能力的方法。在视觉指令微调中，我们在MSCOCO上取得了157.5 CIDEr的新最优图像描述性能，并在ScienceQA上取得了与具有更多参数和广泛视觉-语言预训练的先进模型相当的性能（88.39%）。
英文摘要：
Existing methods to fine-tune LLMs, like Adapter, Prefix-tuning, and LoRA, which introduce extra modules or additional input sequences to inject new skills or knowledge, may compromise the innate abilities of LLMs. In this paper, we propose LLaMA-Excitor, a lightweight method that stimulates the LLMs' potential to better follow instructions by gradually paying more attention to worthwhile information. Specifically, the LLaMA-Excitor does not directly change the intermediate hidden state during the self-attention calculation of the transformer structure. We designed the Excitor block as a bypass module for the similarity score computation in LLMs' self-attention to reconstruct keys and change the importance of values by learnable prompts. LLaMA-Excitor ensures a self-adaptive allocation of additional attention to input instructions, thus effectively preserving LLMs' pre-trained knowledge when fine-tuning LLMs on low-quality instruction-following datasets. Furthermore, we unify the modeling of multi-modal tuning and language-only tuning, extending LLaMA-Excitor to a powerful visual instruction follower without the need for complex multi-modal alignment. Our proposed approach is evaluated in language-only and multi-modal tuning experimental scenarios. Notably, LLaMA-Excitor is the only method that maintains basic capabilities while achieving a significant improvement (+6%) on the MMLU benchmark. In the visual instruction tuning, we achieve a new state-of-the-art image captioning performance of 157.5 CIDEr on MSCOCO, and a comparable performance (88.39%) on ScienceQA to cutting-edge models with more parameters and extensive vision-language pertaining.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
