arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2610.35385cs.RO
收藏

EmbodiedLab:对“为知而行动”的具身智能体进行基准测试

EmbodiedLab: Beyond Manipulation, Benchmarking Embodied Agents That Act to Know

HexaFuture Inc.(赫克萨未来公司);Tsinghua University(清华大学);National University of Singapore(新加坡国立大学);Peking University(北京大学)
Hongcheng Gao, Zelin Zheng, Alex Zhou, Jay Zhu, Shijia Ge, Yazhe Wang, Xiping Li, Jianshu Zeng, Di Wu, Xuan Shangguan, Yexing Wan, Lingyu He, Yongli Wang, Xiaoh… 展开作者

Hongcheng Gao, Zelin Zheng, Alex Zhou, Jay Zhu, Shijia Ge, Yazhe Wang, Xiping Li, Jianshu Zeng, Di Wu, Xuan Shangguan, Yexing Wan, Lingyu He, Yongli Wang, Xiaohan Yu, Zhiqi Jia, Yue Liu, Bryan Hooi, Yinpeng Dong, Wentao Zhang, Sihang Wu, Xiao He

AI总结:

本研究推出具身智能体基准EmbodiedLab,要求机器人为知而行动,评估发现操作技能对弱智能体帮助更大,最强智能体原生控制成功率不足一半。

摘要:

具身智能体需通过自主决定如何及何时与周围环境交互来完成现实世界目标。对于许多此类目标,改变世界的物理状态是手段而非目的:智能体通过交互改变系统状态、观察反馈并进行探索,直至自身达成目标(例如发现物理定律)或学习到目标所需内容;它为知而行动。操作基准仅停留在“行动以改变”阶段,目标状态已给定;科学发现基准则评估“认知”,但用符号操作或数值查询替代物理实验。我们推出EmbodiedLab,一个要求机器人为知而行动的基准。定量科学实验使该设置可控:每个实例都会重新采样系统参数,因此结果无法预先知晓,但可精确评分。在五个学科的30个实验、14个仪器平台上,智能体接收目标和仪器,自行选择操作流程,运行基于物理的模拟实验室,通过摄像头读取仪器数据。评估依据智能体在回合中记录的读数及任务所需的物理状态,检验其认知成果;所有实验均经验证可解。我们在开发者的编码智能体中评估了六个前沿模型,分别测试其原生观测与控制、添加感知与运动规划、以及从最强智能体中蒸馏出的操作技能的表现。采用原生控制时,即使是最强智能体的成功率也不足一半;对于每个智能体,多数失败源于与仪器的交互,且操作技能对较弱智能体的帮助大于感知与规划。该基准及其评估代码可在该httpsURL获取。

英文摘要:

Physical action not only changes the state of an environment but is also the way to know its hidden properties: what cannot be seen must be inferred from the feedback to the change. As embodied agents advance from executing procedures to pursuing goals, success depends on properties neither given nor visible, so they must act to know. Existing benchmarks separate changing from knowing: manipulation benchmarks give the target state, so nothing has to be found out, whereas exploration and discovery benchmarks let the agent find out by looking or querying, so nothing physical has to be changed. Quantitative science experiments offer a natural testbed: the properties sought are new in every instance, appear only in instrument readings under conditions the agent sets up, and can be checked exactly. We introduce EmbodiedLab, a simulated laboratory of 35 such experiments. Given only a goal, the agent must decide what to measure, operate the apparatus with a robot arm, read instruments through cameras and judge when its readings suffice. Its 30 determination tasks specify what to find out, whereas its 5 optimization tasks ask only for the best result the apparatus allows. With native control, the strongest of seven frontier agents succeeds on 41.4% of the tasks. Ablations reveal two bottlenecks: contact-rich manipulation, where ground-truth depth and motion planning change the success rate by at most 10 points, whereas executing every operation for the agent on five of the tasks raises it from at most 20% to at least 70%, so what an agent can know is bounded by how reliably it acts; and closed-loop experimental rigor, where with manipulation supplied, agents fail to gather sufficient readings or read instruments accurately, and most optimization runs stop at one design. Beyond evaluation, EmbodiedLab offers a resamplable environment with verifiable rewards for training embodied agents.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑