通过局部分支路由实现高效且可训练的语言模型测试时扩展
Efficient and Trainable Language Model Test-Time Scaling via Local Branch Routing
AI总结:
提出局部分支路由(LBR)框架,通过轻量级路由器选择局部前瞻树中的深度-1子路径,实现令牌级测试时扩展,在数学推理基准上优于链式思维和软令牌分支基线。
摘要:
测试时扩展改进了语言模型的推理能力,但现有方法往往面临困难的权衡:长链式思维采样仍然是单线程的,而句子级或解决方案级搜索可能计算成本高昂且难以端到端训练。我们引入了局部分支路由(LBR),一种令牌级测试时扩展框架,它扩展一个小型局部前瞻树,将所有采样的分支通过语言模型前向传播,并使用轻量级路由器选择要提交的深度-1子树。通过在候选局部未来的隐藏状态上进行路由,LBR允许每个令牌决策使用超出根节点下一个令牌分布的证据,同时避免完整的解决方案级搜索。由此产生的剪枝-移位-增长解码过程保留了离散分支身份,并定义了一个可处理的树轨迹似然:新增长的节点在首次采样时被计数,路由器决策被分配明确的概率。这使得能够通过可验证奖励进行端到端强化学习,在相同的似然比原则下联合优化基础模型和路由器,如同离散令牌RLVR。在合成层次规划任务上,LBR表明候选后隐藏状态提供了有用的路由证据。在数学推理基准上,LBR在Pass@1和Pass@32上均优于离散链式思维、普通离散令牌RLVR和RL兼容的软令牌分支基线。这些结果表明,轻量级局部分支为语言模型测试时扩展提供了一种高效、可训练且离散的形式。
英文摘要:
Test-time scaling improves language-model reasoning, but existing approaches often face a difficult trade-off: long chain-of-thought sampling remains single-threaded, while sentence- or solution-level search can be computationally expensive and hard to train end-to-end. We introduce Local Branch Routing (LBR), a token-level test-time scaling framework that expands a small local lookahead tree, forwards all sampled branches through the language model, and uses a lightweight router to select the depth-1 subtree to commit. By routing over the hidden states of candidate local futures, LBR allows each token decision to use evidence beyond the root next-token distribution while avoiding full solution-level search. The resulting prune-shift-grow decoding process preserves discrete branch identities and defines a tractable tree-trajectory likelihood: newly grown nodes are counted when first sampled, and router decisions are assigned explicit probabilities. This enables end-to-end reinforcement learning with verifiable rewards, jointly optimizing the base model and router under the same likelihood-ratio principle as discrete-token RLVR. On synthetic hierarchical-planning tasks, LBR shows that post-candidate hidden states provide useful routing evidence. On mathematical reasoning benchmarks, LBR improves both Pass@1 and Pass@32 over discrete chain-of-thought, vanilla discrete-token RLVR, and RL-compatible soft-token branching baselines. These results suggest that lightweight local branching offers an efficient, trainable, and discrete form of language-model test-time scaling.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。