arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2310.00898cs.CL
收藏

使语言模型能够隐式学习自我改进

Enabling Language Models to Implicitly Learn Self-Improvement

University of Illinois Urbana-Champaign(伊利诺伊大学厄巴纳-香槟分校);Google(谷歌公司)

Ziqi Wang, Le Hou, Tianjian Lu, Yuexin Wu, Yunxuan Li, Hongkun Yu, Heng Ji

AI总结:

针对大语言模型自我改进需手动提供明确评分规则的难题,提出隐式自我改进框架PIT,重构RLHF训练目标,从人类偏好数据中隐式学习改进目标,实验证明其效果显著优于基于提示的方法。

摘要:

大语言模型(LLMs)已在开放式文本生成任务中展现出卓越的能力。然而,这类任务固有的开放性意味着模型响应的质量始终存在提升空间。为应对这一挑战,学界已提出多种方法来增强LLMs的性能。目前,人们越来越关注让LLMs能够自我改进其响应质量,从而减少对收集多样化高质量训练数据所需的大量人工标注工作的依赖。近年来,基于提示的方法凭借其有效性、高效性和便捷性,在自我改进方法中得到了广泛探索。不过,这类方法通常需要将明确且详尽编写的评分规则作为输入提供给LLMs。针对现实世界中复杂的改进目标(例如更具帮助性且危害性更低),手动推导并提供所有必要的评分规则成本高昂且颇具挑战。为此,我们提出了ImPlicit Self-ImprovemenT(PIT,隐式自我改进)框架,该框架可从人类偏好数据中隐式学习改进目标。PIT仅需用于训练奖励模型的偏好数据,无需额外的人工投入。具体而言,我们重新构建了基于人类反馈的强化学习(RLHF)的训练目标——我们并非针对给定输入最大化响应质量,而是以参考响应为条件,最大化响应的质量差距。通过这种方式,PIT在“更好地契合人类偏好”的改进目标下被隐式训练。在两个真实世界数据集和一个合成数据集上开展的实验表明,我们的方法性能显著优于基于提示的方法。

英文摘要:

Large Language Models (LLMs) have demonstrated remarkable capabilities in open-ended text generation tasks. However, the inherent open-ended nature of these tasks implies that there is always room for improvement in the quality of model responses. To address this challenge, various approaches have been proposed to enhance the performance of LLMs. There has been a growing focus on enabling LLMs to self-improve their response quality, thereby reducing the reliance on extensive human annotation efforts for collecting diverse and high-quality training data. Recently, prompting-based methods have been widely explored among self-improvement methods owing to their effectiveness, efficiency, and convenience. However, those methods usually require explicitly and thoroughly written rubrics as inputs to LLMs. It is expensive and challenging to manually derive and provide all necessary rubrics with a real-world complex goal for improvement (e.g., being more helpful and less harmful). To this end, we propose an ImPlicit Self-ImprovemenT (PIT) framework that implicitly learns the improvement goal from human preference data. PIT only requires preference data that are used to train reward models without extra human efforts. Specifically, we reformulate the training objective of reinforcement learning from human feedback (RLHF) -- instead of maximizing response quality for a given input, we maximize the quality gap of the response conditioned on a reference response. In this way, PIT is implicitly trained with the improvement goal of better aligning with human preferences. Experiments on two real-world datasets and one synthetic dataset show that our method significantly outperforms prompting-based methods.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑