arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2406.01563cs.CL
收藏

LoFiT: 针对大语言模型表征的局部微调

LoFiT: Localized Fine-tuning on LLM Representations

The University of Texas at Austin(德克萨斯大学奥斯汀分校);Princeton University(普林斯顿大学)

Fangcong Yin, Xi Ye, Greg Durrett

AI总结:

本研究提出LoFiT局部微调框架,通过选定任务相关的稀疏注意力头并训练偏移向量实现LLM适配,效果优于传统表征干预方法,且以远少于LoRA的参数量达到相当性能。

摘要:

近期可解释性领域的研究表明,大语言模型(LLM)可通过无学习的方式适配新任务:通过干预LLM的表征即可引导模型产生符合对齐要求的预期行为。例如,已有研究报道称向特定注意力头的输出添加特定偏置向量可提升模型的真实性。\n在本研究中,我们证实局部微调是这类表征干预方法的有效替代方案。我们提出了名为“针对LLM表征的局部微调(Localized Fine-Tuning on LLM Representations, LoFiT)”的框架,该框架先识别对学习特定任务最重要的注意力头子集,随后训练偏移向量,将其添加到模型在这些选定注意力头处的隐藏表征中。LoFiT仅定位到稀疏的注意力头集合(占比3%-10%),并基于有限训练数据学习偏移向量,其设置与表征干预方法所用的条件相当。\n在真实性任务和推理任务上,我们发现LoFiT的干预向量对LLM适配的效果优于推理时干预(Inference-time Intervention)等表征干预方法生成的向量。我们还发现局部定位步骤十分关键:选择任务特定的注意力头集合,相比干预为其他任务选定的注意力头,能带来更优的性能。最后,在我们研究的7项任务中,尽管LoFiT修改的参数量比LoRA等其他参数高效微调方法少20倍至200倍,其性能却与这些方法相当。

英文摘要:

Recent work in interpretability shows that large language models (LLMs) can be adapted for new tasks in a learning-free way: it is possible to intervene on LLM representations to elicit desired behaviors for alignment. For instance, adding certain bias vectors to the outputs of certain attention heads is reported to boost the truthfulness of models. In this work, we show that localized fine-tuning serves as an effective alternative to such representation intervention methods. We introduce a framework called Localized Fine-Tuning on LLM Representations (LoFiT), which identifies a subset of attention heads that are most important for learning a specific task, then trains offset vectors to add to the model's hidden representations at those selected heads. LoFiT localizes to a sparse set of heads (3%-10%) and learns the offset vectors from limited training data, comparable to the settings used for representation intervention. For truthfulness and reasoning tasks, we find that LoFiT's intervention vectors are more effective for LLM adaptation than vectors from representation intervention methods such as Inference-time Intervention. We also find that the localization step is important: selecting a task-specific set of attention heads can lead to higher performance than intervening on heads selected for a different task. Finally, across 7 tasks we study, LoFiT achieves comparable performance to other parameter-efficient fine-tuning methods such as LoRA, despite modifying 20x-200x fewer parameters than these methods.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑