通过克隆实现通用学习委托
General learned delegation by clones
AI总结:
SELFCEST通过代理强化学习实现通用学习委托,利用克隆技术在不同并行上下文中分配资源,提升数学推理和长上下文问答任务的准确性与效率。
摘要:
前沿语言模型通过增加测试时的计算量来提高性能,但固定推理预算下,串行推理或无协调的并行采样可能效率低下。我们提出了SELFCEST,通过代理强化学习使基础模型能够生成具有相同权重的克隆,在独立的并行上下文中运行。训练过程采用全局任务奖励进行端到端训练,利用共享参数的回放,产生一个能够跨分支分配生成和上下文预算的学习控制器。在具有挑战性的数学推理基准和长上下文多跳问答任务中,SELFCEST在匹配的推理预算下,相对于单一模型基线改进了准确性-成本帕累托前沿,并在两个领域展示了分布外泛化能力。
英文摘要:
Frontier language models improve with additional test-time computation, but serial reasoning or uncoordinated parallel sampling can be compute-inefficient under fixed inference budgets. We propose SELFCEST, which equips a base model with the ability to spawn same-weight clones in separate parallel contexts by agentic reinforcement learning. Training is end-to-end under a global task reward with shared-parameter rollouts, yielding a learned controller that allocates both generation and context budget across branches. Across challenging math reasoning benchmarks and long-context multi-hop QA, SELFCEST improves the accuracy-cost Pareto frontier relative to monolithic baselines at matched inference budget, and exhibits out-of-distribution generalization in both domains.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。