结合直接偏好优化的自训练提升思维链推理能力
Self-Training with Direct Preference Optimization Improves Chain-of-Thought Reasoning
AI总结：
本文提出将直接偏好优化（DPO）集成到自训练中，利用模型自身输出的偏好数据引导其生成更准确多样的思维链推理，从而提升小规模语言模型在数学推理任务上的性能，同时降低对大型专有模型的依赖与成本。
摘要：
为了在数学推理任务上有效训练语言模型（LMs），需要高质量的监督微调数据。除了从人类专家获取标注外，一种常见的替代方案是从更大、更强大的语言模型中采样。然而，这种知识蒸馏方法可能成本高昂且不稳定，尤其是在依赖如 GPT-4 这类闭源、专有语言模型时，其行为往往难以预测。在本研究中，我们证明了小规模语言模型的推理能力可以通过自训练得到增强，即模型从自身的输出中学习的过程。我们还表明，传统的自训练可以通过一种名为 Direct Preference Optimization（DPO，直接偏好优化）的偏好学习算法进一步增强。通过将 DPO 集成到自训练中，我们利用偏好数据引导语言模型生成更准确、更多样化的思维链推理。我们使用不同的基础模型在各种数学推理任务上评估了该方法。实验表明，该方法不仅提升了语言模型的推理性能，而且与依赖大型专有语言模型相比，提供了一种更具成本效益和可扩展性的解决方案。
英文摘要：
Effective training of language models (LMs) for mathematical reasoning tasks demands high-quality supervised fine-tuning data. Besides obtaining annotations from human experts, a common alternative is sampling from larger and more powerful LMs. However, this knowledge distillation approach can be costly and unstable, particularly when relying on closed-source, proprietary LMs like GPT-4, whose behaviors are often unpredictable. In this work, we demonstrate that the reasoning abilities of small-scale LMs can be enhanced through self-training, a process where models learn from their own outputs. We also show that the conventional self-training can be further augmented by a preference learning algorithm called Direct Preference Optimization (DPO). By integrating DPO into self-training, we leverage preference data to guide LMs towards more accurate and diverse chain-of-thought reasoning. We evaluate our method across various mathematical reasoning tasks using different base models. Our experiments show that this approach not only improves LMs' reasoning performance but also offers a more cost-effective and scalable solution compared to relying on large proprietary LMs.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
