arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2409.15684cs.RO
收藏

SYNERGAI:面向人机协作的感知对齐

SYNERGAI: Perception Alignment for Human-Robot Collaboration

Beijing Institute for General Artificial Intelligence (BIGAI)(北京通用人工智能研究院)

Yixin Chen, Guoxi Zhang, Yaowei Zhang, Hongming Xu, Peiyuan Zhi, Qing Li, Siyuan Huang

AI总结:

提出SYNERGAI系统,利用3D场景图作为显式表示,结合大型语言模型分解任务并分配工具,通过在线交互更新场景图自动纠正人机感知错位,在真实场景中显著提升了任务对齐与执行的成功率。

摘要:

近期,大型语言模型(LLM)在促进人机交互与协作方面展现出强大潜力。然而,现有的基于LLM的系统往往忽视了人类与机器人感知之间的错位,这阻碍了它们的有效通信和真实世界机器人部署。为解决此问题,我们引入了SYNERGAI,一个旨在同时实现感知对齐和人机协作的统一系统。其核心是,SYNERGAI采用3D场景图(3DSG)作为其显式且固有的表示。这使系统能够利用LLM分解复杂任务,并在中间步骤中分配适当的工具,以从3DSG中提取相关信息、修改其结构或生成响应。重要的是,SYNERGAI结合了一种自动机制,通过在线交互更新其3DSG,从而实现与用户的感知错位纠正。SYNERGAI在ScanQA中以零样本方式取得了与数据驱动模型相当的性能。通过在10个真实世界场景中的全面实验,SYNERGAI证明了其在与人类建立共识方面的有效性,在对齐任务中实现了61.9%的成功率。它还通过转移在对齐过程中获取的知识,将新任务的成功率从3.7%显著提升至45.68%。

英文摘要:

Recently, large language models (LLMs) have shown strong potential in facilitating human-robotic interaction and collaboration. However, existing LLM-based systems often overlook the misalignment between human and robot perceptions, which hinders their effective communication and real-world robot deployment. To address this issue, we introduce SYNERGAI, a unified system designed to achieve both perceptual alignment and human-robot collaboration. At its core, SYNERGAI employs 3D Scene Graph (3DSG) as its explicit and innate representation. This enables the system to leverage LLM to break down complex tasks and allocate appropriate tools in intermediate steps to extract relevant information from the 3DSG, modify its structure, or generate responses. Importantly, SYNERGAI incorporates an automatic mechanism that enables perceptual misalignment correction with users by updating its 3DSG with online interaction. SYNERGAI achieves comparable performance with the data-driven models in ScanQA in a zero-shot manner. Through comprehensive experiments across 10 real-world scenes, SYNERGAI demonstrates its effectiveness in establishing common ground with humans, realizing a success rate of 61.9% in alignment tasks. It also significantly improves the success rate from 3.7% to 45.68% on novel tasks by transferring the knowledge acquired during alignment.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑