arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2405.21040cs.CLcs.AI
收藏

通过质量感知自精炼直接对齐语言模型

Direct Alignment of Language Models via Quality-Aware Self-Refinement

Hong Kong University of Science and Technology(香港科技大学);LightSpeed Studios, Tencent(腾讯光速工作室);Centre for Artificial Intelligence and Robotics, Hong Kong Institute of Science & Innovation, CAS(中国科学院香港创新研究院人工智能与机器人中心)

Runsheng Yu, Yong Wang, Xiaoqi Jiao, Youzhi Zhang, James T. Kwok

AI总结:

针对DPO忽略正负响应相对质量的问题,提出利用LLM内在知识设计精炼函数估计响应质量并自精炼损失函数,将其集成至DPO和IPO中,有效提升了微调模型的性能。

摘要:

人类反馈强化学习(RLHF)通常被用于使大型语言模型(LLMs)的行为与人类偏好对齐。最近,一种流行的替代方案是直接策略优化(DPO),它用策略本身替代了基于LLM的奖励模型,从而免去了学习奖励模型所需的额外内存和训练时间。然而,DPO没有考虑正负响应的相对质量,可能导致次优的训练结果。为了缓解这一问题,我们研究了利用即时微调LLM的内在知识来获取相对质量,并帮助精炼损失函数。具体而言,我们利用LLM的知识设计了一个精炼函数,用于估计正负响应的质量。我们证明了在温和的假设下,构建的精炼函数能够帮助自精炼损失函数。该精炼函数被集成到DPO及其变体恒等策略优化(IPO)中。在各种评估器上的实验表明,它们能够提升微调模型相对于DPO和IPO的性能。

英文摘要:

Reinforcement Learning from Human Feedback (RLHF) has been commonly used to align the behaviors of Large Language Models (LLMs) with human preferences. Recently, a popular alternative is Direct Policy Optimization (DPO), which replaces an LLM-based reward model with the policy itself, thus obviating the need for extra memory and training time to learn the reward model. However, DPO does not consider the relative qualities of the positive and negative responses, and can lead to sub-optimal training outcomes. To alleviate this problem, we investigate the use of intrinsic knowledge within the on-the-fly fine-tuning LLM to obtain relative qualities and help to refine the loss function. Specifically, we leverage the knowledge of the LLM to design a refinement function to estimate the quality of both the positive and negative responses. We show that the constructed refinement function can help self-refine the loss function under mild assumptions. The refinement function is integrated into DPO and its variant Identity Policy Optimization (IPO). Experiments across various evaluators indicate that they can improve the performance of the fine-tuned models over DPO and IPO.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑