arXiv 2606.119092026-06-11cs.AI

Embodied-BenchClaw：用于具身空间智能基准构建的自主多智能体系统

Embodied-BenchClaw: An Autonomous Multi-Agent System for Embodied Spatial Intelligence Benchmark Construction

QiYuan Lab（启元实验室）；School of Information and Software Engineering, University of Electronic Science and Technology of China（电子科技大学信息与软件工程学院）；Beijing University of Posts and Telecommunications（北京邮电大学）；School of Computer Science and Engineering, Northeastern University（东北大学计算机科学与工程学院）；School of Computer Science and Engineering, Beihang University（北京航空航天大学计算机科学与工程学院）

Baoyang Jiang, Fengchun Zhang, Leyuan Wang, Haotian Li, Yida Wang, Zhe Ji, Jinshan Lai, Xi Ren, Jianwei Hu, Qiang Ma

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

提出Embodied-BenchClaw，一个通过五阶段流水线和三个智能体协调的自主系统，自动构建可验证、可执行、可维护且诊断有用的具身空间智能基准，减少人工工作量。

摘要：

基准测试对于评估具身空间智能至关重要，但其构建劳动密集、难以重用且维护困难。现有的具身基准通常是静态的，随着模型改进可能迅速饱和，限制其区分新能力的能力。我们提出Embodied-BenchClaw，一个用于构建具身空间智能基准的自主智能体系统。给定用户指定的评估意图，Embodied-BenchClaw通过五个阶段流水线自动生成完整且可持续更新的基准包：意图蓝图、数据收集、结构化与清洗、基准合成、评估报告。该流水线由三个智能体协调：规划、构建和评估。为提高可重用性和可靠性，Embodied-BenchClaw引入了可扩展的技能库和过程质量控制，使基准构建可组合、可验证和可修复。我们实例化了多个基准，涵盖室内空间推理、室外空间推理、机器人操作、四足机器人导航、无人机/空中视图理解以及静态基准增强。这些基准跨越不同的具身载体、数据源和空间能力。通过人工评估、基于评判者的评估、一致性检查、成本分析和消融实验，结果表明Embodied-BenchClaw能够以较少的人工努力构建可验证、可执行、可维护且诊断有用的具身空间基准。

英文摘要：

Benchmarks are essential for evaluating embodied spatial intelligence, yet their construction is labor-intensive, hard to reuse, and difficult to maintain. Existing embodied benchmarks are often static and may quickly become saturated as models improve, limiting their ability to distinguish new capabilities. We propose Embodied-BenchClaw, an autonomous agentic system for constructing embodied spatial intelligence benchmarks. Given a user-specified evaluation intent, Embodied-BenchClaw automatically produces a complete and continually updatable benchmark package through a five-stage pipeline: intent blueprinting, data collection, structuring and cleaning, benchmark synthesis, and evaluation reporting. The pipeline is coordinated by three agents for planning, construction, and evaluation. To improve reusability and reliability, Embodied-BenchClaw introduces an extensible Skill Library and process quality control, enabling benchmark construction to be composable, verifiable, and repairable. We instantiate multiple benchmarks covering indoor spatial reasoning, outdoor spatial reasoning, robotic manipulation, quadruped robot navigation, UAV/aerial-view understanding, and static benchmark enhancement. These benchmarks span diverse embodied carriers, data sources, and spatial capabilities. Experiments with human evaluation, judge-based assessment, consistency checks, cost analysis, and ablations show that Embodied-BenchClaw can construct verifiable, executable, maintainable, and diagnostically useful embodied spatial benchmarks with reduced manual effort.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。