arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2610.23946cs.LGcs.CL
收藏

智能体强化学习中的数据缩放规律:定律与合成策略

How Data Scales in Agentic Reinforcement Learning: Laws and Synthesis Strategies

MBZUAI(穆罕默德·本·扎耶德人工智能大学);McGill University(麦吉尔大学);CityUHK(香港城市大学);Columbia University(哥伦比亚大学)

Bowei He, Yankai Chen, Xiaokun Zhang, Changjiang Han, Ye Yuan, Chong Li, Xue Liu

AI总结:

该研究针对智能体RL的数据缩放问题,分解数据维度并拟合缩放定律,推导预算分配的最优方案,使计算效率较主流方法最高提升1.7倍。

摘要:

预训练缩放定律将训练数据视为一维量:token 数量。智能体强化学习(RL)继承了这种标量抽象,但现代合成流程现在可独立生成任务、环境和轨迹,且成本差异极大,这使数据转变为结构化设计空间。这提出了预训练从未需回答的问题:给定合成预算,应缩放哪个维度?我们通过三个紧密关联的贡献端到端解决该问题。首先,我们将智能体数据分解为三个基数维度(任务、环境、轨迹),加上奖励密度控制变量,并在数学+代码、SWE、网页及UI/工具应用领域的不同大语言模型(LLM)上拟合各维度的缩放关系。在所有测量的模型规模、模型家族、领域及RL算法(PPO和GRPO)中,缩放指数顺序为环境>任务>轨迹。当将环境多样性分解为世界和环境结构多样性时,其优势主要由前者驱动,而后者与任务多样性相当。其次,我们将拟合定律作为通用标尺为合成方法定价,将各方法的效率捕获为各维度的折扣因子,并确定开始崩溃的关键合成-真实比率;这些比率在各维度间相差一个数量级。第三,我们将预算分配视为在我们的定律和折扣下的约束优化,推导了Chinchilla风格的计算最优方案,并在保留的规模上验证了它。实证结果表明,遵循该方案可使计算效率比主流的“缩放轨迹”实践最高提升1.7倍。

英文摘要:

Pretraining scaling laws treat training data as a one-dimensional quantity: a token count. Agentic reinforcement learning (RL) inherits this scalar abstraction, but modern synthesis pipelines now generate \textit{tasks}, \textit{environments}, and \textit{trajectories} independently and at very different costs, turning data into a structured design space. This raises a question pretraining never had to answer: given a synthesis budget, \textit{which axis} should be scaled? We address this question end-to-end with three tightly coupled contributions. First, we decompose agentic data into three cardinality axes (task, environment, trajectory), plus a reward-density control variable, and fit per-axis scaling relations on different large language models (LLMs) across math+code, SWE, web, and UI/tool application domains. Across all model sizes, model families, domains, and RL algorithms (PPO and GRPO) we measured, the scaling exponents order as environment $>$ task $>$ trajectory. When environment diversity is decomposed into world and environment-structure diversity, its advantage is mainly driven by the former, while the latter is just comparable to the task diversity. Second, we use the fitted laws as a common ruler to price synthesis methods, capturing each method's efficiency as a per-axis discount factor and identifying critical synthetic-to-real ratios at which collapse begins; these ratios differ by an order of magnitude across axes. Third, we cast budget allocation as a constrained optimization under our laws and discounts, deriving a Chinchilla-style compute-optimal recipe and validating it at held-out scale. The empirical results demonstrate that following the recipe yields up to $1.7\times$ compute efficiency over the dominant ``scale-the-trajectories'' practice.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑