Recently, large language models (LLMs) have shown strong potential in facilitating human-robotic interaction and collaboration. However, existing LLM-based systems often overlook the misalignment between human and robot perceptions, which hinders their effective communication and real-world robot deployment. To address this issue, we introduce SYNERGAI, a unified system designed to achieve both perceptual alignment and human-robot collaboration. At its core, SYNERGAI employs 3D Scene Graph (3DSG) as its explicit and innate representation. This enables the system to leverage LLM to break down complex tasks and allocate appropriate tools in intermediate steps to extract relevant information from the 3DSG, modify its structure, or generate responses. Importantly, SYNERGAI incorporates an automatic mechanism that enables perceptual misalignment correction with users by updating its 3DSG with online interaction. SYNERGAI achieves comparable performance with the data-driven models in ScanQA in a zero-shot manner. Through comprehensive experiments across 10 real-world scenes, SYNERGAI demonstrates its effectiveness in establishing common ground with humans, realizing a success rate of 61.9% in alignment tasks. It also significantly improves the success rate from 3.7% to 45.68% on novel tasks by transferring the knowledge acquired during alignment.
Transforming Monolithic Foundation Models into Embodied Multi-Agent Architectures for Human-Robot Collaboration
Department of Computer Science and Technology, Tsinghua University(计算机科学与技术系,清华大学);School of Artificial Intelligence, Beijing University of Posts and Telecommunications(人工智能学院,北京邮电大学)
Show Me What You Can Do: Capability Calibration on Reachable Workspace for Human-Robot Collaboration
UCLA Center for Vision, Cognition, Learning, and Autonomy (VCLA)(加州大学洛杉矶分校视觉、认知、学习与自主中心(VCLA));Massachusetts Institute of Technology(麻省理工学院);UCLA Department of Psychology(加州大学洛杉矶分校心理学系);Beijing Institute for General Artificial Intelligence (BIGAI)(北京通用人工智能研究院(BIGAI));Peking University(北京大学);Tsinghua University(清华大学)
Australian Institute for Machine Learning(澳大利亚机器学习研究所);Andy Thomas Centre for Space Resources(安迪·托马斯空间资源中心);The University of Adelaide(阿德莱德大学);SmartSat CRC(智能卫星合作研究中心)
Risk Occupancy: A New and Efficient Paradigm through Vehicle-Road-Cloud Collaboration
School of Vehicle and Mobility, Tsinghua University(清华大学车辆与出行学院);State Key Laboratory of Intelligent Green Vehicle and Mobility, Tsinghua University(清华大学智能绿色车辆与出行国家重点实验室);University of Wisconsin-Madison(威斯康星大学麦迪逊分校)
MA-CoNav: A Master-Slave Multi-Agent Framework with Hierarchical Collaboration and Dual-Level Reflection for Long-Horizon Embodied VLN
Southwestern University of Finance(西南财经大学);Department of Informatics,Universitat Hamburg, Hamburg, Germany(汉堡大学信息学院);University of Electronic Science(电子科学大学)
CoPeD-Advancing Multi-Robot Collaborative Perception: A Comprehensive Dataset in Real-World Environments
New York University(纽约大学);Tandon School of Engineering(坦登工程学院);U.S. Army Combat Capabilities Development Command(美国陆军作战能力发展司令部);Army Research Laboratory(陆军研究实验室)
Not All Uncertainty Matters: Simulation-in-the-Loop Fast-Slow Reasoning for Decision-Critical Autonomous Driving System
Shenzhen Research Institute of Big Data(深圳大数据研究院);Shenzhen Institutes of Advanced Technology (SIAT), Chinese Academy of Sciences(中国科学院深圳先进技术研究院);The University of New South Wales(新南威尔士大学);University of Macau(澳门大学);The University of Hong Kong(香港大学)
Heterogeneous Robot Collaboration in Unstructured Environments with Grounded Generative Intelligence
Dept. of Electrical and Computer Engineering at Texas A&M University(德克萨斯A&M大学电气与计算机工程系);GRASP Laboratory at the University of Pennsylvania(宾夕法尼亚大学GRASP实验室)
LPAC: Learnable Perception-Action-Communication Loops with Applications to Coverage Control
GRASP Laboratory, University of Pennsylvania(宾夕法尼亚大学GRASP实验室);Massachusetts Institute of Technology(麻省理工学院);University of Pennsylvania(宾夕法尼亚大学);Department of Electrical and Systems Engineering, University of Pennsylvania(宾夕法尼亚大学电气与系统工程系)