超越拒绝采样:轨迹融合用于扩展数学推理
Beyond Rejection Sampling: Trajectory Fusion for Scaling Mathematical Reasoning
AI总结:
TrajFusion通过融合错误与正确推理轨迹,改进数学推理微调,提升复杂问题表现。
摘要:
大型语言模型(LLMs)在数学推理方面取得了显著进展,通常通过拒绝采样进行微调,仅保留正确的推理轨迹。虽然有效,但这种范式将监督视为二元过滤器,系统性地排除教师生成的错误,导致训练过程中推理失败如何建模存在空白。在本文中,我们提出TrajFusion,一种微调策略,将拒绝采样重新诠释为结构化监督构建过程。具体而言,TrajFusion生成融合轨迹,通过将所选错误轨迹与反思提示和正确轨迹交错,显式建模试错推理。每个融合样本的长度根据教师错误的频率和多样性进行自适应控制,在挑战性问题上提供更丰富的监督,同时在错误信号不具信息性时安全地退化为传统的拒绝采样微调(RFT)。TrajFusion不需要更改架构或训练目标。在多个数学基准上的广泛实验表明,TrajFusion在挑战性和长形式推理问题上 consistently 超过RFT。
英文摘要:
Large language models (LLMs) have made impressive strides in mathematical reasoning, often fine-tuned using rejection sampling that retains only correct reasoning trajectories. While effective, this paradigm treats supervision as a binary filter that systematically excludes teacher-generated errors, leaving a gap in how reasoning failures are modeled during training. In this paper, we propose TrajFusion, a fine-tuning strategy that reframes rejection sampling as a structured supervision construction process. Specifically, TrajFusion forms fused trajectories that explicitly model trial-and-error reasoning by interleaving selected incorrect trajectories with reflection prompts and correct trajectories. The length of each fused sample is adaptively controlled based on the frequency and diversity of teacher errors, providing richer supervision for challenging problems while safely reducing to vanilla rejection sampling fine-tuning (RFT) when error signals are uninformative. TrajFusion requires no changes to the architecture or training objective. Extensive experiments across multiple math benchmarks demonstrate that TrajFusion consistently outperforms RFT, particularly on challenging and long-form reasoning problems.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。