arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2410.07513cs.LGcs.AIcs.CL
收藏

用于语言模型对齐的演化对比蒸馏

Evolutionary Contrastive Distillation for Language Model Alignment

Amazon(亚马逊公司)

Julian Katz-Samuels, Zheng Li, Hyokun Yun, Priyanka Nigam, Yi Xu, Vaclav Petricek, Bing Yin, Trishul Chilimbi

AI总结:

提出演化对比蒸馏(ECD)方法,通过逐步演化指令生成困难负样本对比数据,结合DPO对比学习算法提升语言模型复杂指令遵循能力,7B模型性能超越SOTA 7B并媲美70B模型。

摘要:

大型语言模型(LLMs)执行复杂指令的能力对其现实世界应用至关重要。然而,近期几项研究表明LLMs在处理具有挑战性的指令时存在困难。本文提出演化对比蒸馏(ECD),一种用于生成高质量合成偏好数据的新方法,旨在增强语言模型遵循复杂指令的能力。ECD生成的数据专门揭示成功遵循一组复杂指令的响应与高质量但存在细微错误的响应之间的差异。该方法通过提示LLMs将简单指令逐步演化为复杂指令来实现。当指令复杂度增加时,针对原指令的成功响应成为新指令的“困难负样本”响应,基本满足新指令要求,但仅差一两点。通过将优质响应与此类困难负样本配对,并采用DPO等对比学习算法,我们提升了语言模型遵循复杂指令的能力。实验表明,该方法生成的7B模型超越当前SOTA 7B模型的复杂指令遵循性能,甚至可与开源70B模型竞争。

英文摘要:

The ability of large language models (LLMs) to execute complex instructions is essential for their real-world applications. However, several recent studies indicate that LLMs struggle with challenging instructions. In this paper, we propose Evolutionary Contrastive Distillation (ECD), a novel method for generating high-quality synthetic preference data designed to enhance the complex instruction-following capability of language models. ECD generates data that specifically illustrates the difference between a response that successfully follows a set of complex instructions and a response that is high-quality, but nevertheless makes some subtle mistakes. This is done by prompting LLMs to progressively evolve simple instructions to more complex instructions. When the complexity of an instruction is increased, the original successful response to the original instruction becomes a "hard negative" response for the new instruction, mostly meeting requirements of the new instruction, but barely missing one or two. By pairing a good response with such a hard negative response, and employing contrastive learning algorithms such as DPO, we improve language models' ability to follow complex instructions. Empirically, we observe that our method yields a 7B model that exceeds the complex instruction-following performance of current SOTA 7B models and is competitive even with open-source 70B models.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑