arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2606.25354cs.CL
收藏

通过局部分支路由实现高效且可训练的语言模型测试时扩展

Efficient and Trainable Language Model Test-Time Scaling via Local Branch Routing

Northwestern University(西北大学);Rutgers University(罗格斯大学);University of Wisconsin–Madison(威斯康星大学麦迪逊分校);Carnegie Mellon University(卡内基梅隆大学);LMSYS Org(LMSYS组织);Tilde Research(Tilde研究);University of California, Santa Barbara(加州大学圣塔芭芭拉分校);University of Toronto(多伦多大学);University of British Columbia(不列颠哥伦比亚大学);University of California, San Diego(加州大学圣迭戈分校)
Yutong Yin, Mingyu Jin, Jin Pan, Changyi Yang, Zijie Xia, Dhruv Pai, Shuming Hu, Zhen Zhang, Chenyang Zhao, Jinman Zhao, Wujiang Xu, Raymond Li, Xin Eric Wang, … 展开作者

Yutong Yin, Mingyu Jin, Jin Pan, Changyi Yang, Zijie Xia, Dhruv Pai, Shuming Hu, Zhen Zhang, Chenyang Zhao, Jinman Zhao, Wujiang Xu, Raymond Li, Xin Eric Wang, Julian McAuley, Zhaoran Wang

AI总结:

提出局部分支路由(LBR)框架,通过轻量级路由器选择局部前瞻树中的深度-1子路径,实现令牌级测试时扩展,在数学推理基准上优于链式思维和软令牌分支基线。

摘要:

测试时扩展改进了语言模型的推理能力,但现有方法往往面临困难的权衡:长链式思维采样仍然是单线程的,而句子级或解决方案级搜索可能计算成本高昂且难以端到端训练。我们引入了局部分支路由(LBR),一种令牌级测试时扩展框架,它扩展一个小型局部前瞻树,将所有采样的分支通过语言模型前向传播,并使用轻量级路由器选择要提交的深度-1子树。通过在候选局部未来的隐藏状态上进行路由,LBR允许每个令牌决策使用超出根节点下一个令牌分布的证据,同时避免完整的解决方案级搜索。由此产生的剪枝-移位-增长解码过程保留了离散分支身份,并定义了一个可处理的树轨迹似然:新增长的节点在首次采样时被计数,路由器决策被分配明确的概率。这使得能够通过可验证奖励进行端到端强化学习,在相同的似然比原则下联合优化基础模型和路由器,如同离散令牌RLVR。在合成层次规划任务上,LBR表明候选后隐藏状态提供了有用的路由证据。在数学推理基准上,LBR在Pass@1和Pass@32上均优于离散链式思维、普通离散令牌RLVR和RL兼容的软令牌分支基线。这些结果表明,轻量级局部分支为语言模型测试时扩展提供了一种高效、可训练且离散的形式。

英文摘要:

Test-time scaling improves language-model reasoning, but existing approaches often face a difficult trade-off: long chain-of-thought sampling remains single-threaded, while sentence- or solution-level search can be computationally expensive and hard to train end-to-end. We introduce Local Branch Routing (LBR), a token-level test-time scaling framework that expands a small local lookahead tree, forwards all sampled branches through the language model, and uses a lightweight router to select the depth-1 subtree to commit. By routing over the hidden states of candidate local futures, LBR allows each token decision to use evidence beyond the root next-token distribution while avoiding full solution-level search. The resulting prune-shift-grow decoding process preserves discrete branch identities and defines a tractable tree-trajectory likelihood: newly grown nodes are counted when first sampled, and router decisions are assigned explicit probabilities. This enables end-to-end reinforcement learning with verifiable rewards, jointly optimizing the base model and router under the same likelihood-ratio principle as discrete-token RLVR. On synthetic hierarchical-planning tasks, LBR shows that post-candidate hidden states provide useful routing evidence. On mathematical reasoning benchmarks, LBR improves both Pass@1 and Pass@32 over discrete chain-of-thought, vanilla discrete-token RLVR, and RL-compatible soft-token branching baselines. These results suggest that lightweight local branching offers an efficient, trainable, and discrete form of language-model test-time scaling.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑