拒绝的理由？用评判对齐语言模型
Reasons to Reject? Aligning Language Models with Judgments
AI总结：
本文首次系统探索用自然语言评判而非标量奖励对齐大语言模型，提出对比非似然训练框架CUT，仅用1317条评判数据即显著超越基线，证明评判在对齐中比奖励更具潜力。
摘要：
作为人类，我们不断与同伴互动，并以自然语言的形式接收反馈。这种语言反馈使我们能够保持适当的行为，并纠正潜在的错误。一个自然产生的问题是：我们能否利用语言反馈来对齐大语言模型（LLMs）？与以往使用标量奖励来对齐LLMs的研究不同，我们首次从语言反馈（即评判）的视角对对齐进行了系统性探索。我们首先深入考察了可用于使LLMs与评判对齐的潜在方法，发现这些方法无法充分利用评判信息。为了更有效地利用评判，我们提出了一种新颖的框架——对比非似然训练（CUT），该框架能够基于评判实现细粒度的不当内容检测与纠正。实验结果表明，仅使用1317条现成的评判数据，CUT（LLaMA2-13b）就能击败175B参数的DaVinci003，并在AlpacaEval上比最佳基线高出50.84分。CUT（LLaMA2-chat-13b）还可以利用最新的模型特定评判以迭代方式对齐LLMs，在AlpacaEval上的性能从81.09分提升至91.68分。进一步分析表明，在LLM对齐中，评判比奖励具有更大的潜力。
英文摘要：
As humans, we consistently interact with our peers and receive feedback in the form of natural language. This language feedback allows us to maintain appropriate behavior, and rectify potential errors. The question arises naturally: can we use language feedback to align large language models (LLMs)? In contrast to previous research that aligns LLMs with scalar rewards, we present the first systematic exploration of alignment through the lens of language feedback (i.e., judgment). We start with an in-depth investigation of potential methods that can be adapted for aligning LLMs with judgments, revealing that these methods cannot fully capitalize on judgments. To facilitate more effective utilization of judgments, we propose a novel framework, Contrastive Unlikelihood Training (CUT), that allows for fine-grained inappropriate content detection and correction based on judgments. Our results show that, with merely 1317 off-the-shelf judgment data, CUT (LLaMA2-13b) can beat the 175B DaVinci003 and surpass the best baseline by 50.84 points on AlpacaEval. CUT (LLaMA2-chat-13b) can also align LLMs in an iterative fashion using up-to-date model-specific judgments, improving performance from 81.09 to 91.68 points on AlpacaEval. Further analysis suggests that judgments hold greater potential than rewards in LLM alignment.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
