大语言模型中潜在推理的高效训练后优化
Efficient Post-Training Refinement of Latent Reasoning in Large Language Models
AI总结：
针对大语言模型潜在推理训练后更新难题，提出含对比推理反馈与残差嵌入优化的轻量级框架，优化潜在推理轨迹，在5个推理基准验证有效性，MathQA准确率无额外训练提升5%。
摘要：
推理是大语言模型语言理解的关键组成部分。尽管思维链提示通过显式中间步骤提升性能，但存在显著的token开销和固定推理轨迹问题，阻碍了分步优化。近期潜在推理研究通过直接在模型潜在空间优化内部推理过程、不生成显式输出，解决了这些局限性。然而，关键挑战仍存：训练后如何有效更新推理嵌入以引导模型得到更准确的解决方案。为应对该挑战，我们提出轻量级训练后框架，通过两种新策略优化潜在推理轨迹：1）对比推理反馈，将推理嵌入与强弱基线对比，通过嵌入增强推断有效更新方向；2）残差嵌入优化，通过逐步整合当前与历史梯度稳定更新，实现快速且可控的收敛。我们在5个推理基准上开展大量实验与案例研究，验证了所提框架的有效性，其中在MathQA上无额外训练即可获得5%的准确率提升。
英文摘要：
Reasoning is a key component of language understanding in Large Language Models. While Chain-of-Thought prompting enhances performance via explicit intermediate steps, it suffers from sufficient token overhead and a fixed reasoning trajectory, preventing step-wise refinement. Recent advances in latent reasoning address these limitations by refining internal reasoning processes directly in the model's latent space, without producing explicit outputs. However, a key challenge remains: how to effectively update reasoning embeddings during post-training to guide the model toward more accurate solutions. To overcome this challenge, we propose a lightweight post-training framework that refines latent reasoning trajectories using two novel strategies: 1) Contrastive reasoning feedback, which compares reasoning embeddings against strong and weak baselines to infer effective update directions via embedding enhancement; 2) Residual embedding refinement, which stabilizes updates by progressively integrating current and historical gradients, enabling fast yet controlled convergence. Extensive experiments and case studies are conducted on five reasoning benchmarks to demonstrate the effectiveness of the proposed framework. Notably, a 5\% accuracy gain on MathQA without additional training.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
