LongDPO：通过批评增强的逐步信息解锁大语言模型更优的长文本生成能力
LongDPO: Unlock Better Long-form Generation Abilities for LLMs via Critique-augmented Stepwise Information
AI总结：
针对现有大模型长文本生成质量不佳、偏好学习缺乏长上下文细粒度反馈的问题，提出LongDPO方法，结合过程监督、蒙特卡洛树搜索、外部批评优化逐步偏好对，实现长文本生成效果提升且通用性能几乎无损。
摘要：
长文本生成对于学术论文写作和仓库级代码生成至关重要。尽管如此，包括GPT-4o在内的当前模型性能仍不尽如人意。现有采用结果监督偏好学习的方法往往无法为长上下文提供详细反馈，这一缺陷会导致生成内容无法完全满足查询要求，出现长度偏差、质量下降等问题。本文提出通过引入过程监督来提升长文本生成效果。我们使用蒙特卡洛树搜索（Monte Carlo Tree Search）收集逐步偏好对，借助全局内存池维持一致性。为解决候选选择不佳的问题，我们整合外部批评来优化并提升偏好对质量。最后，我们利用收集到的逐步偏好对执行步骤级DPO（直接偏好优化）。实验结果表明，该方法在长文本生成基准上提升了生成长度与质量，且在不同模型骨干网络的通用基准上性能几乎无损。
英文摘要：
Long-form generation is crucial for academic writing papers and repo-level code generation. Despite this, current models, including GPT-4o, still exhibit unsatisfactory performance. Existing methods that utilize preference learning with outcome supervision often fail to provide detailed feedback for extended contexts. This shortcoming can lead to content that does not fully satisfy query requirements, resulting in issues like length deviations, and diminished quality. In this paper, we propose enhancing long-form generation by incorporating process supervision. We employ Monte Carlo Tree Search to gather stepwise preference pairs, utilizing a global memory pool to maintain consistency. To address the issue of suboptimal candidate selection, we integrate external critiques to refine and improve the quality of the preference pairs. Finally, we apply step-level DPO using the collected stepwise preference pairs. Experimental results show that our method improves length and quality on long-form generation benchmarks, with almost lossless performance on general benchmarks across various model backbones.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
