arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2610.18878cs.LGcs.AIstat.ML
收藏

用于仅答案数据的高效后训练与验证的逆推理模型

Inverse Reasoning Models for Efficient Post-Training and Verification from Answer-Only Data

Striveworks

Eric O. Korman

AI总结:

本研究提出利用仅答案数据训练逆推理LLM的自蒸馏技术,可高效后训练推理LLM,提升困难提示的pass@1,还可作为验证器提升选择准确率,在Countdown任务中效果显著。

摘要:

后训练推理类大语言模型(LLM)常需在高成本监督与计算低效间权衡:监督微调通常需要推理轨迹,而在许多应用中仅拥有提示-答案元组、缺乏真实思维链(CoT)序列,因此数据效率低下。另一方面,强化学习(例如带有可验证奖励)可在无CoT数据的情况下运行,但计算效率极低,因为其每优化步骤需要多个自回归完成结果,且无法保证任何结果能提供非零策略梯度。本研究探索了一种中间方案,即利用仅答案数据的自蒸馏技术,将CoT视为潜在变量。该方案依赖于训练一个配套的“逆推理LLM”,其在给定答案的条件下生成原模型的推理轨迹。逆模型仅需训练一次,即可在后续仅答案数据批次中重复使用,所有生成过程均离线且提前完成,使该方法特别适合持续后训练。在Countdown和GSM8K任务上使用Qwen3-0.6B模型,我们发现GRPO训练饱和后,来自逆模型的单次离线蒸馏大致使困难提示的pass@1翻倍;在Countdown任务中,其在所有混合比例下的所有困难尾部指标上均优于拒绝采样自蒸馏(在GSM8K上增益较小),而拒绝采样对于模型自身从未解决的问题天然无信号。在新数据上继续GRPO训练在困难pass@1上仍保持优势,但在pass@8和pass@32上落后于蒸馏,且每一步都需重新生成rollout。此外,逆模型还有第二项应用:作为准确的正确性验证器,在测试时用于n选最佳选择时,其将Countdown任务的选择准确率从0.753(多数投票)提升至0.969。

英文摘要:

Post-training reasoning LLMs often requires making a tradeoff between costly supervision and compute inefficiency: supervised finetuning typically requires reasoning traces, which is data inefficient since in many applications one only has prompt-answer tuples and lacks the groundtruth chain-of-thought (CoT) sequence. On the other hand, reinforcement learning (with verifiable rewards, for example) can operate without CoT data but it is much more compute-inefficient since it requires multiple autoregressive completions per optimization step, none of which are guaranteed to give a non-zero policy gradient. In this work we explore a middleground, a self-distillation technique that uses answer-only data, treating CoT as latent variables. This relies on training a partner ``inverse reasoning LLM" that generates the original's reasoning traces conditioned on the answer. The inverse model is trained once and reused across incoming batches of answer-only data, with all generation happening offline and up front, making the approach particularly suited to continual post-training. Using Qwen3-0.6B on the Countdown and GSM8K tasks, we show that after GRPO training has saturated, a single offline distillation round from the inverse model roughly doubles pass@1 on hard prompts and, on Countdown, outperforms rejection-sampling self-distillation on every hard-tail metric at every mix ratio (with smaller gains on GSM8K); rejection sampling by construction has no signal on problems the model never solves on its own. Continued GRPO on the fresh data retains an edge at hard pass@1 but trails distillation at pass@8 and pass@32 while regenerating rollouts at every step. As an added bonus we see that the inverse model has a second application as an accurate verifier of correctness. When used for best-of-$n$ selection at test time, it raises selection accuracy on Countdown from 0.753 (majority voting) to 0.969.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑