AI 中文总结
针对多步智能体扁平训练忽视可复用子过程的问题，提出从数据中构建可复用经验树X-Tree并集成到三种训练设置，在三个基准上显著提升成功率。
AI 中文摘要
多步智能体在扁平动作流上训练：SFT和RLVR对每个词元一视同仁，忽略了跨任务重复出现的子过程，以及让人类能够从可复用例程自上而下规划的层级结构。这种结构未被利用，扁平训练对每条稀缺轨迹的利用程度低于其内容所允许的水平。最近的智能体确实使用了这种结构，但仅作为上下文中由LLM编写的技能，从未体现在权重中，因此其收益无法超越检索实现泛化。我们则从数据本身恢复这种层级结构并基于其训练，无需任何LLM调用。遵循文本分词器仅通过计数构建词表的方法，我们根据可复用性对动作片段评分，并将规范化动作合并为可复用的经验树（X-Tree）。每个X-Tree节点捕获频繁且成功导向的技能如何由子技能组成，从而指导高效泛化。我们将X-Tree集成到三种训练设置中：离线RL，每个节点作为一个训练实例；在线RLVR，带有自适应技能奖励；以及同策略自蒸馏，以X-Tree作为自教师特权上下文。在WebArena、ScienceWorld和WebShop上，跨三个模型规模，在匹配数据和预算条件下，X-Tree相比标准方案在WebArena上提升高达4.5%成功率，在ScienceWorld上提升5.8%成功率，在WebShop上提升4.1%成功率。匹配分析将收益归因于X-Tree结构和三种集成方式。
英文摘要
Multi-step agents are trained on flat action streams: SFT and RLVR weight every token uniformly and ignore the sub-procedures that recur across tasks, the hierarchy that lets humans plan top-down from reusable routines. This structure sits unused, and flat training uses each scarce trajectory less fully than its content allows. Recent agents do use that structure, but only as LLM-written skills in context, never in the weights, so their gains do not generalize beyond retrieval. We instead recover this hierarchy from the data itself and train on it, with no LLM calls. Following text tokenizers, which build a vocabulary by counting alone, we score action spans by reusability and merge canonicalized actions into a reusable eXperience tree (X-Tree). Each X-Tree node captures how a frequent and success-bearing skill is composed from sub-skills, guiding efficient generalization. We integrate X-Tree into three training settings: offline RL, with each node as a training instance; online RLVR, with an adaptive skill bonus; and on-policy self-distillation, with X-Tree as the self-teacher's privileged context. Across WebArena, ScienceWorld, and WebShop at three model scales, X-Tree improves over standard recipes at matched data and budget by up to 4.5% SR on WebArena, 5.8% SR on ScienceWorld and 4.1% success on WebShop. Matched analyses attribute the gains to the X-Tree structure and the three integrations.
CommentsProject in Progress. Homepage: https://sitaocheng.github.io/xtree/. Code: https://github.com/sitaocheng/X-Tree/
