辩论、训练、进化：语言模型推理的自进化
DEBATE, TRAIN, EVOLVE: Self Evolution of Language Model Reasoning
AI总结：
针对大语言模型仅靠数据提升推理能力不现实的问题，提出无需真实标注的DTE自进化框架及Reflect-Critique-Refine提示策略，实验证明其在多推理基准上准确率提升显著，且具备强跨域泛化性。
摘要：
大语言模型（LLMs）通过在海量数据集上的广泛训练，其推理能力已得到显著提升。然而，仅依赖额外数据来实现提升正变得越来越不切实际，这凸显了让模型在无需外部监督的情况下自主增强推理能力的必要性。本文提出了Debate, Train, Evolve（DTE，辩论-训练-进化），这是一种无需真实标注的新型训练框架，利用多智能体辩论轨迹来进化单个语言模型。我们还提出了一种新的提示策略Reflect-Critique-Refine（反思-批判-优化），通过明确指示智能体批判并优化自身推理过程来提升辩论质量。我们使用六个开源权重模型在七个推理基准上开展了广泛评估，结果显示DTE框架实现了大幅提升，在具有挑战性的GSM-PLUS数据集上平均准确率提升了8.92%。此外，我们观察到该方法具备强跨域泛化性，在所有其他基准上的平均准确率提升了5.8%，表明我们的方法捕捉到了通用推理能力。我们的框架代码和训练好的模型已在https://github.com/ctrl-gaurav/Debate-Train-Evolve公开。
英文摘要：
Large language models (LLMs) have improved significantly in their reasoning through extensive training on massive datasets. However, relying solely on additional data for improvement is becoming increasingly impractical, highlighting the need for models to autonomously enhance their reasoning without external supervision. In this paper, we propose Debate, Train, Evolve (DTE), a novel ground truth-free training framework that uses multi-agent debate traces to evolve a single language model. We also introduce a new prompting strategy Reflect-Critique-Refine, to improve debate quality by explicitly instructing agents to critique and refine their reasoning. Extensive evaluations on seven reasoning benchmarks with six open-weight models show that our DTE framework achieve substantial improvements, with an average accuracy gain of 8.92% on the challenging GSM-PLUS dataset. Furthermore, we observe strong cross-domain generalization, with an average accuracy gain of 5.8% on all other benchmarks, suggesting that our method captures general reasoning capabilities. Our framework code and trained models are publicly available at https://github.com/ctrl-gaurav/Debate-Train-Evolve
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
