arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2511.02130cs.AIcs.LG
收藏

Re-FORC:用于高效思维链推理的自适应奖励预测

Re-FORC: Adaptive Reward Prediction for Efficient Chain-of-Thought Reasoning

AWS Agentic AI(AWS智能代理部门);Carnegie Mellon University(卡内基梅隆大学)

Renos Zabounidis, Aditya Golatkar, Michael Kleinman, Alessandro Achille, Wei Xia, Stefano Soatto

AI总结:

研究提出Re-FORC自适应奖励预测方法,通过在推理模型上训练轻量级适配器,实现早期停止无前景推理链、优化模型和思维长度选择以及自适应测试时缩放,有效提升推理效率和准确率。

摘要:

我们提出了Re-FORC,一种自适应奖励预测方法。给定一个查询,它能根据未来思维令牌数量预测预期未来奖励。Re-FORC在推理模型上训练一个轻量级适配器,在更长推理和更大模型上展示出改进的预测。它能实现:1)早期停止无前景的推理链,与固定预算截止相比计算量最多减少26%且保持准确率;2)优化模型和思维长度选择,峰值准确率比单独最大模型高1.7个百分点且计算量最多少12%;3)自适应测试时缩放,比基于置信度的基线平均提高9.9个百分点。Re-FORC允许通过令牌成本阈值进行长度控制的动态推理并预先估计计算时间。

英文摘要:

We propose Re-FORC, an adaptive reward prediction method that, given a query, enables prediction of the expected future rewards as a function of the number of future thinking tokens. Re-FORC trains a lightweight adapter on reasoning models, demonstrating improved prediction with longer reasoning and larger models. Re-FORC enables: 1) early stopping of unpromising reasoning chains, reducing compute by up to 26\% compared to fixed-budget cutoffs, while maintaining accuracy, 2) optimized model and thinking length selection that outperforms the largest model alone--- reaching 1.7 percentage points higher peak accuracy while needing up to 12\% less compute to match the largest model's accuracy, 3) adaptive test-time scaling, which increases accuracy by 9.9 percentage points (on average at maximum compute) over confidence-based baselines. Re-FORC allows dynamic reasoning with length control via cost-per-token thresholds while estimating computation time upfront.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑