TARo: 令牌级自适应路由用于大语言模型测试时间对齐
TARo: Token-level Adaptive Routing for LLM Test-time Alignment
AI总结：
TARo通过令牌级自适应路由在推理时引导冻结的LLM进行结构化推理，提升推理性能达22.4%，并在医疗和指令遵循任务中表现优异。
摘要：
大型语言模型（LLMs）具有强大的推理能力，但通常需要昂贵的后训练来达到高性能。最近的测试时间对齐方法提供了轻量级替代方案，但主要集中在偏好对齐而非推理上。为此，我们提出令牌级自适应路由（TARo），在推理时完全引导冻结的LLM进行结构化推理。具体而言，我们首先在逐步数学轨迹上训练奖励模型以捕捉细粒度的逻辑一致性信号，然后引入一个可学习的令牌级路由器，自动控制奖励模型对基础模型的指导。大量实验表明，TARo在基础模型上推理性能提升达22.4%，在现有令牌级测试时间对齐方法上提升8.4%，同时提升医疗领域推理（MedXpertQA）和指令遵循（AlpacaEval）。此外，TARo还能从小型到大型骨干网络泛化，无需重新训练，将测试时间对齐从偏好优化扩展到鲁棒的跨领域推理。
英文摘要：
Large language models (LLMs) exhibit strong reasoning capabilities but typically require expensive post-training to reach high performance. Recent test-time alignment methods offer a lightweight alternative, but have been explored mainly for preference alignment rather than reasoning. To bridge this gap, we propose, Token-level Adaptive Routing (TARo), which steers frozen LLMs toward structured reasoning entirely at inference time. Specifically, we first train reward models on step-wise mathematical traces to capture fine-grained logical consistency signals, then introduce a learnable token-level router that automatically controls the guidance of the reward model to the base model. Extensive experiments show that TARo significantly improves reasoning performance by up to +22.4% over base model and +8.4% over existing token-level test-time alignment methods, while also boosting out-of-distribution clinical reasoning (MedXpertQA) and instruction following (AlpacaEval). Furthermore, TARo also generalizes from small to large backbones without retraining, extending test-time alignment from preference optimization to robust, cross-domain reasoning.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
