基于过程的自奖励语言模型
Process-based Self-Rewarding Language Models
AI总结：
针对现有自奖励方法在数学推理中效果不佳的问题，提出基于过程的自奖励流程，引入长思维推理、逐步LLM评判与逐步偏好优化，显著提升了模型在多个数学推理基准上的性能。
摘要：
大型语言模型在各类下游任务中展现出卓越性能，并广泛应用于多种场景。人类标注的偏好数据被用于训练以进一步提升LLMs性能，但这受限于人类能力的上限。因此，自奖励方法被提出，即LLMs通过奖励自身输出来生成训练数据。然而，现有的自奖励范式在数学推理场景中效果不佳，甚至可能导致性能下降。在本研究中，我们提出了针对语言模型的基于过程的自奖励流程，在自奖励范式中引入了长思维推理、逐步LLM作为评判者和逐步偏好优化。通过迭代的基于过程的自奖励，我们的新范式成功提升了LLMs在多个数学推理基准上的性能，展示了自奖励在实现可能超越人类能力的LLM推理方面的巨大潜力。
英文摘要：
Large Language Models have demonstrated outstanding performance across various downstream tasks and have been widely applied in multiple scenarios. Human-annotated preference data is used for training to further improve LLMs' performance, which is constrained by the upper limit of human performance. Therefore, Self-Rewarding method has been proposed, where LLMs generate training data by rewarding their own outputs. However, the existing self-rewarding paradigm is not effective in mathematical reasoning scenarios and may even lead to a decline in performance. In this work, we propose the Process-based Self-Rewarding pipeline for language models, which introduces long-thought reasoning, step-wise LLM-as-a-Judge, and step-wise preference optimization within the self-rewarding paradigm. Our new paradigm successfully enhances the performance of LLMs on multiple mathematical reasoning benchmarks through iterative Process-based Self-Rewarding, demonstrating the immense potential of self-rewarding to achieve LLM reasoning that may surpass human capabilities.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
