训练语言模型进行批判以实现更好的优化
Training Language Model to Critique for Better Refinement
AI总结：
本文提出面向优化的批判优化（RCO）框架，通过利用优化信号训练批判模型，以批判效用作为奖励信号指导执行者模型优化响应，显著提升了LLM在多任务中的批判质量与优化效果。
摘要：
大型语言模型（LLMs）展现出了卓越的评估和批判能力，能够提供有洞察力的反馈并识别各种任务中的缺陷。然而，关于哪种类型的批判对改善模型响应最有效，或如何生成此类批判，目前的研究探索仍然有限。为了填补这一空白，我们引入了面向优化的批判优化（RCO），这是一种旨在利用优化信号训练批判模型的新颖框架。RCO采用了一个反馈循环，其中由批判模型生成的批判指导执行者模型优化其响应。批判效用（CU）量化了这些优化的有效性，作为训练批判模型的奖励信号。通过专注于能够带来更好优化的批判，RCO消除了对直接批判偏好评估的需求，确保了那些驱动有意义改进的批判得到奖励。我们在五个任务上评估了RCO，即对话生成、摘要生成、问答、数学推理和代码生成，并表明它在批判质量和优化结果方面显著优于传统方法和开源模型。我们的贡献包括引入RCO、一种基于优化响应偏好的新颖监督方案，以及突显该方法在增强LLM批判-优化循环方面有效性的全面实验结果。
英文摘要：
Large language models (LLMs) have demonstrated remarkable evaluation and critique capabilities, providing insightful feedback and identifying flaws in various tasks. However, limited research has explored which types of critiques are most effective for improving model responses or how to generate such critiques. To address this gap, we introduce \textbf{R}efinement-oriented \textbf{C}ritique \textbf{O}ptimization (RCO), a novel framework designed to train critic models using refinement signals. RCO uses a feedback loop where critiques, generated by the critic model, guide the actor model in refining its responses. The critique utility (CU) quantifies the effectiveness of these refinements, serving as the reward signal for training the critic model. By focusing on critiques that lead to better refinements, RCO eliminates the need for direct critique preference assessment, ensuring that critiques driving meaningful improvements are rewarded. We evaluate RCO across five tasks, i.e., dialog generation, summarization, question answering, mathematical reasoning, and code generation, and show that it significantly outperforms traditional methods and open-source models in terms of critique quality and refinement outcomes. Our contributions include the introduction of RCO, a novel supervision scheme based on refined response preferences, and comprehensive experimental results that highlight the method's effectiveness in enhancing LLM critique-refinement loops.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
