arXiv 2409.139482025-02-23cs.CL

使用后续回答似然作为奖励信号对齐语言模型

Aligning Language Models Using Follow-up Likelihood as Reward Signal

National University of Singapore（新加坡国立大学）；Peking University（北京大学）；The Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳））；Tencent AI Lab（腾讯AI实验室）；Shenzhen Research Institute of Big Data（深圳大数据研究院）；University of Science and Technology of China（中国科学技术大学）

Chen Zhang, Dading Chong, Feng Jiang, Chengguang Tang, Anningzhe Gao, Guohua Tang, Haizhou Li

备注：: Accepted by AAAI-2025, 16 pages, reward model, LLM Alignment, code repository at (https://github.com/e0397123/FLR)

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

本文提出利用用户后续话语的似然作为奖励信号（FLR），无需人工或商业LLM偏好标注即可区分回答优劣，在多个基准上媲美强奖励模型，并用于自动挖掘偏好数据以提升基础模型有用性。

摘要：

在自然的人与人对话中，参与者通常会根据对方的后续反应获得反馈信号。这些反应包括口头回答、面部表情、情绪状态的变化以及其他非语言线索。类似地，在人机交互中，机器可以利用用户的后续话语作为反馈信号，以评估其是否恰当满足了用户的请求。因此，我们提出使用后续话语的似然作为奖励，以区分更受偏好的回答与不太受偏好的回答，而无需依赖人工或基于商业大语言模型的偏好标注。我们提出的奖励机制——"后续似然作为奖励"（FLR）——在8个成对偏好基准和4个评分基准上，其性能可与在大规模人工或GPT-4标注数据上训练的强奖励模型相媲美。基于FLR机制，我们提出从基础策略模型的在线生成中自动挖掘偏好数据。这些偏好数据随后通过直接对齐偏好（DAP）方法（如直接偏好优化（DPO））用于提升基础模型的有用性。最后，我们证明，使用自然语言反馈对提供后续似然的语言模型进行微调，能显著提升FLR在奖励建模基准上的性能，并增强其在对齐基础策略模型有用性方面的有效性。

英文摘要：

In natural human-to-human conversations, participants often receive feedback signals from one another based on their follow-up reactions. These reactions can include verbal responses, facial expressions, changes in emotional state, and other non-verbal cues. Similarly, in human-machine interactions, the machine can leverage the user's follow-up utterances as feedback signals to assess whether it has appropriately addressed the user's request. Therefore, we propose using the likelihood of follow-up utterances as rewards to differentiate preferred responses from less favored ones, without relying on human or commercial LLM-based preference annotations. Our proposed reward mechanism, ``Follow-up Likelihood as Reward" (FLR), matches the performance of strong reward models trained on large-scale human or GPT-4 annotated data on 8 pairwise-preference and 4 rating-based benchmarks. Building upon the FLR mechanism, we propose to automatically mine preference data from the online generations of a base policy model. The preference data are subsequently used to boost the helpfulness of the base model through direct alignment from preference (DAP) methods, such as direct preference optimization (DPO). Lastly, we demonstrate that fine-tuning the language model that provides follow-up likelihood with natural language feedback significantly enhances FLR's performance on reward modeling benchmarks and effectiveness in aligning the base policy model's helpfulness.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。