arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2606.09138cs.LGcs.CL
收藏

Claw-R1:面向智能体强化学习的步骤级数据中间件系统

AgentWorld: A Data Factory for Training LLM Agents from Multi-Turn Interactions

State Key Laboratory of Cognitive Intelligence, University of Science and Technology of China(中国科学技术大学认知智能国家重点实验室)

Daoyu Wang, Mingyue Cheng, Qingchuan Li, Shuo Yu, Jie Ouyang, Qi Liu, Enhong Chen

AI总结:

提出Claw-R1系统,通过网关服务器和数据池组件,将智能体交互步骤转化为结构化数据资产,支持实时检查、质量筛选和训练批次配置,解决智能体强化学习中数据生命周期管理问题。

摘要:

智能体强化学习已成为将大语言模型从静态聊天机器人转变为交互式智能体的重要后训练范式,催生了如OpenClaw等代表性应用。现有工作主要关注策略优化算法和训练框架,但对从数据产生到训练消费的智能体-环境交互完整数据生命周期关注不足。为弥补这一差距,我们提出Claw-R1,一个面向智能体强化学习的交互式步骤级数据中间件系统。Claw-R1通过两个核心组件——网关服务器和数据池——连接异构智能体运行时与强化学习训练后端。网关服务器通过统一的LLM API入口捕获多轮交互步骤,而数据池将其组织为由提示ID、响应ID、奖励和其他元数据组成的步骤级记录。在我们的演示中,用户可以交互式检查实时轨迹,查看每一步的状态、动作和奖励,根据质量和就绪程度筛选数据,并为不同的下游强化学习算法配置训练就绪批次。总体而言,Claw-R1将智能体交互轨迹视为受管理的数据资产,而非临时运行时日志。通过此演示,我们希望鼓励社区认识到数据管理在智能体强化学习中的重要性。我们的代码可在https://github.com/AgentR1/Claw-R1获取,演示视频可在https://youtu.be/Pw47dAOw6B0找到。

英文摘要:

Training LLMs to act as capable agents requires learning from multi-turn interactions with environments. However, multi-turn interaction trajectories generated by heterogeneous agent runtimes are difficult to transform into structured, high-quality training data that training engines can readily consume. In this paper, we propose AgentWorld, a data factory that bridges heterogeneous agent runtimes and training engines through a Gateway Server and a Data Pool, reframing LLM agent training around a unified data lifecycle from interaction collection to training consumption. The Gateway Server captures multi-turn interactions through an LLM API proxy, while the Data Pool organizes them into standardized step records for curation and training. An interactive dashboard supports trajectory monitoring, step inspection, and data review. In our demonstration, users integrate ready-to-use white-box and black-box agents, trace and curate their interaction data across the pipeline, and extend the factory to new agents and datasets through lightweight configuration. Quantitative experiments across multiple datasets and RL methods demonstrate training support for different agent runtimes. AgentWorld provides a practical foundation for transforming heterogeneous multi-turn experience into training-ready data for LLM agents. Our code is available at https://github.com/AgentR1/Claw-R1 and the demonstration video can be found at link https://youtu.be/Pw47dAOw6B0.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑