arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2602.13262cs.AIcs.CL
收藏

通过克隆实现通用学习委托

General learned delegation by clones

Qiuzhen College, Tsinghua University(清华大学齐振学院);Pattern Recognition Center, WeChat AI, Tencent Inc(微信AI模式识别中心,腾讯公司)

Darren Li, Meiqi Chen, Chenze Shao, Fandong Meng, Jie Zhou

AI总结:

SELFCEST通过代理强化学习实现通用学习委托,利用克隆技术在不同并行上下文中分配资源,提升数学推理和长上下文问答任务的准确性与效率。

摘要:

前沿语言模型通过增加测试时的计算量来提高性能,但固定推理预算下,串行推理或无协调的并行采样可能效率低下。我们提出了SELFCEST,通过代理强化学习使基础模型能够生成具有相同权重的克隆,在独立的并行上下文中运行。训练过程采用全局任务奖励进行端到端训练,利用共享参数的回放,产生一个能够跨分支分配生成和上下文预算的学习控制器。在具有挑战性的数学推理基准和长上下文多跳问答任务中,SELFCEST在匹配的推理预算下,相对于单一模型基线改进了准确性-成本帕累托前沿,并在两个领域展示了分布外泛化能力。

英文摘要:

Frontier language models improve with additional test-time computation, but serial reasoning or uncoordinated parallel sampling can be compute-inefficient under fixed inference budgets. We propose SELFCEST, which equips a base model with the ability to spawn same-weight clones in separate parallel contexts by agentic reinforcement learning. Training is end-to-end under a global task reward with shared-parameter rollouts, yielding a learned controller that allocates both generation and context budget across branches. Across challenging math reasoning benchmarks and long-context multi-hop QA, SELFCEST improves the accuracy-cost Pareto frontier relative to monolithic baselines at matched inference budget, and exhibits out-of-distribution generalization in both domains.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑