arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2408.10841cs.AIcs.CL
收藏

DELIA:面向大型语言模型指令适应性的多样性增强学习

DELIA: Diversity-Enhanced Learning for Instruction Adaptation in Large Language Models

Beijing University of Posts and Telecommunications(北京邮电大学);Harbin Institute of Technology(哈尔滨工业大学)

Yuanhao Zeng, Fei Ren, Xinpeng Zhou, Yihang Wang, Yingxia Shao

AI总结:

针对指令微调中模型仅适应格式而非获取知识的问题,提出DELIA方法,利用多样性数据缓冲将有偏特征转化为理想特征近似,在翻译和格式化文本生成任务上显著优于基线。

摘要:

尽管指令微调被广泛用于调整大型语言模型(LLMs)的行为,但大量经验证据和研究表明,这主要是一个模型适应特定任务格式的过程,而非获取新知识或能力。我们认为,这一局限性源于指令微调期间学习到的有偏特征,这些特征与理想的任务特定特征不同,导致在下游任务中学习到的底层语义较少。然而,理想特征未知且不可计算,这限制了以往工作依赖先验知识来辅助推理或训练,从而将LLMs的能力限制在开发者的能力范围内,而非数据驱动的可扩展学习。在本文中,通过我们新颖的数据合成方法DELIA(多样性增强的指令适应性学习),我们利用LLMs训练中大量多样化数据的缓冲效应,将指令微调中的有偏特征转化为理想特征的近似,而无需显式的先验理想特征。实验表明,与常见指令微调和其他基线相比,DELIA表现更优。在冰岛语-英语翻译的BLEURT分数上(WMT-21数据集,gemma-7b-it),它比常见指令微调高出17.07%-33.41%,并在格式化文本生成(Llama2-7b-chat)上提高了36.1%的准确率。值得注意的是,在我们已知的知识注入方法中,DELIA独特地将新特殊标记的内部表示与其先验语义对齐。

英文摘要:

Although instruction tuning is widely used to adjust behavior in Large Language Models (LLMs), extensive empirical evidence and research indicates that it is primarily a process where the model fits to specific task formats, rather than acquiring new knowledge or capabilities. We propose that this limitation stems from biased features learned during instruction tuning, which differ from ideal task-specfic features, leading to learn less underlying semantics in downstream tasks. However, ideal features are unknown and incalculable, constraining past work to rely on prior knowledge to assist reasoning or training, which limits LLMs' capabilities to the developers' abilities, rather than data-driven scalable learning. In our paper, through our novel data synthesis method, DELIA (Diversity-Enhanced Learning for Instruction Adaptation), we leverage the buffering effect of extensive diverse data in LLMs training to transform biased features in instruction tuning into approximations of ideal features, without explicit prior ideal features. Experiments show DELIA's better performance compared to common instruction tuning and other baselines. It outperforms common instruction tuning by 17.07%-33.41% on Icelandic-English translation bleurt score (WMT-21 dataset, gemma-7b-it) and improves accuracy by 36.1% on formatted text generation (Llama2-7b-chat). Notably, among knowledge injection methods we've known, DELIA uniquely align the internal representations of new special tokens with their prior semantics.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑