arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2406.07168cs.CL
收藏

通过从语言反馈中学习教语言模型自我改进

Teaching Language Models to Self-Improve by Learning from Language Feedback

NiuTrans Research(NiuTrans研究院)

Chi Hu, Yimin Hu, Hang Cao, Tong Xiao, Jingbo Zhu

AI总结:

本文提出自我精炼调优(SRT)方法,通过让基础模型从更先进模型的语言反馈和精炼结果中学习,实现自我改进,在 AlpacaEval 2.0 上将 70B 模型胜率从 9.6% 提升至 25.8%,超越多个成熟系统。

摘要:

将大语言模型(LLMs)与人类意图和价值观对齐至关重要但颇具挑战。当前方法主要依赖人类偏好,这不仅成本高昂,而且不足以捕捉以自然语言表达的细微反馈。本文提出自我精炼调优(Self-Refinement Tuning, SRT),一种利用模型反馈进行对齐的方法,从而减少对人类标注的依赖。SRT 使用基础语言模型(如 Tulu2)生成初始响应,然后由更先进的模型(如 GPT-4-Turbo)进行批评和精炼。这一过程使基础模型能够自我评估并改进其输出,促进持续学习。SRT 进一步通过从自身生成的反馈和精炼结果中学习来优化模型,形成一个促进模型改进的反馈循环。我们的实证评估表明,SRT 在不同任务和模型规模上显著优于强基线。当应用于 70B 参数模型时,SRT 在 AlpacaEval 2.0 基准上将胜率从 9.6% 提高到 25.8%,超越了 GPT-4-0314、Claude 2 和 Gemini 等成熟系统。我们的分析强调了语言反馈在 SRT 成功中的关键作用,表明该方向具有进一步探索的潜力。

英文摘要:

Aligning Large Language Models (LLMs) with human intentions and values is crucial yet challenging. Current methods primarily rely on human preferences, which are costly and insufficient in capturing nuanced feedback expressed in natural language. In this paper, we present Self-Refinement Tuning (SRT), a method that leverages model feedback for alignment, thereby reducing reliance on human annotations. SRT uses a base language model (e.g., Tulu2) to generate initial responses, which are critiqued and refined by a more advanced model (e.g., GPT-4-Turbo). This process enables the base model to self-evaluate and improve its outputs, facilitating continuous learning. SRT further optimizes the model by learning from its self-generated feedback and refinements, creating a feedback loop that promotes model improvement. Our empirical evaluations demonstrate that SRT significantly outperforms strong baselines across diverse tasks and model sizes. When applied to a 70B parameter model, SRT increases the win rate from 9.6\% to 25.8\% on the AlpacaEval 2.0 benchmark, surpassing well-established systems such as GPT-4-0314, Claude 2, and Gemini. Our analysis highlights the crucial role of language feedback in the success of SRT, suggesting potential for further exploration in this direction.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑