Building Cooperative Embodied Agents Modularly with Large Language Models
University of Massachusetts Amherst（马萨诸塞大学阿默斯特分校）；Tsinghua University（清华大学）；Shanghai Jiao Tong University（上海交通大学）；MIT（麻省理工学院）；MIT-IBM Watson AI Lab（麻省理工学院-IBM沃森人工智能实验室）
Hongxin Zhang, Weihua Du, Jiaming Shan, Qinhong Zhou, Yilun Du, Joshua B. Tenenbaum, Tianmin Shu, Chuang Gan
In this work, we address challenging multi-agent cooperation problems with decentralized control, raw sensory observations, costly communication, and multi-objective tasks instantiated in various embodied environments. While previous research either presupposes a cost-free communication channel or relies on a centralized controller with shared observations, we harness the commonsense knowledge, reasoning ability, language comprehension, and text generation prowess of LLMs and seamlessly incorporate them into a cognitive-inspired modular framework that integrates with perception, memory, and execution. Thus building a Cooperative Embodied Language Agent CoELA, who can plan, communicate, and cooperate with others to accomplish long-horizon tasks efficiently. Our experiments on C-WAH and TDW-MAT demonstrate that CoELA driven by GPT-4 can surpass strong planning-based methods and exhibit emergent effective communication. Though current Open LMs like LLAMA-2 still underperform, we fine-tune a CoELA with data collected with our agents and show how they can achieve promising performance. We also conducted a user study for human-agent interaction and discovered that CoELA communicating in natural language can earn more trust and cooperate more effectively with humans. Our research underscores the potential of LLMs for future research in multi-agent cooperation. Videos can be found on the project website https://vis-www.cs.umass.edu/Co-LLM-Agents/.
Constrained Human-AI Cooperation: An Inclusive Embodied Social Intelligence Challenge
Carnegie Mellon University（卡内基梅隆大学）；Peking University（北京大学）；University of California, Santa Barbara（加州大学圣芭芭拉分校）；Harvard University（哈佛大学）；University of Massachusetts Amherst（马萨诸塞大学阿姆赫斯特分校）；MIT（麻省理工学院）；Johns Hopkins University（约翰霍普金斯大学）；Honda Research Institute USA（本田美国研究院）
How Personas Can Influence Agents to Play Split or Steal
Universidade Lusófona（卢索福纳大学）；Universitat de Barcelona（巴塞罗那大学）；Université Paris Cité（巴黎城市大学）；Computational Neuropsychology and Simulation (CNS) Laboratory（计算神经心理学与模拟（CNS）实验室）；Arizona State University（亚利桑那州立大学）
COMBO: Compositional World Models for Embodied Multi-Agent Cooperation
University of Massachusetts Amherst（马萨诸塞大学阿默斯特分校）；IIIS, Tsinghua University（清华大学信息科学技术学院）；Peking University（北京大学）；Johns Hopkins University（约翰斯·霍普金斯大学）；Honda Research Institute USA（美国本田研究所）；MIT（麻省理工学院）
Communicating Plans, Not Percepts: Scalable Multi-Agent Coordination with Embodied World Models
Department of Computer Science University of Wisconsin-Madison（计算机科学系 明尼苏达大学）；Department of Computer Science National University of Singapore（计算机科学系 新加坡国立大学）
Learn as Individuals, Evolve as a Team: Multi-agent LLMs Adaptation in Embodied Environments
The Hong Kong University of Science and Technology（香港科技大学）；Institute of Artificial Intelligence (TeleAI), China Telecom（中国电信人工智能研究院（TeleAI））；East China University of Science and Technology（华东理工大学）
EARBench: Towards Evaluating Physical Risk Awareness for Task Planning of Foundation Model-based Embodied AI Agents
School of Data Science, The Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳）数据科学学院）；AI Lab, Tencent（腾讯AI实验室）；Robotics X, Tencent（腾讯Robotics X实验室）；Nanjing University of Posts and Telecommunications（南京邮电大学）
From Grunts to Lexicons: Emergent Language from Cooperative Foraging
Institute for Infocomm Research (I 2 {}^{\text{2}} R), A ∗ STAR, Singapore（新加坡信息与通信研究院）；College of Computing and Data Science, Nanyang Technological University, Singapore（南洋理工大学计算机科学与数据科学学院）；Department of Computer Science, The University of Manchester, United Kingdom（曼彻斯特大学计算机科学系）；Centre for Frontier AI Research (CFAR), A ∗ STAR, Singapore（前沿人工智能研究中心）；Sakana AI, Japan（日本Sakana AI）
Large Model Enabled Embodied Intelligence for 6G Integrated Perception, Communication, and Computation Network
School of Information and Electronics（信息与电子学院）；School of Interdisciplinary Science（交叉科学学院）；State Key Laboratory of CNS/ATM（CNS/ATM国家重点实验室）；Beijing Institute of Technology（北京理工大学）；Advanced Technology Research Institute（先进技术研究院）；Yangtze Delta Region Academy（长江三角洲地区学院）；School of School of Information Science and Engineering（信息科学与工程学院）；Institute of Intelligent Communication Technologies（智能通信技术研究院）；Shandong Key Laboratory of Intelligent Communication and Sensing-Computing Integration（智能通信与传感-计算集成山东省重点实验室）；Zhejiang Provincial Key Laboratory of Information Processing, Communication and Networking（信息处理、通信与网络浙江省重点实验室）；Department of Electronic Engineering（电子工程系）；State Key Laboratory of Space Network and Communications（空间网络与通信国家重点实验室）；National Mobile Communications Research Laboratory（移动通信研究中心）；Purple Mountain Laboratories（紫金山实验室）
Modeling Theory of Mind in Multi-Agent Games Using Adaptive Feedback Control
Institute for Bioengineering of Catalonia (IBEC)（加泰罗尼亚生物工程研究所（IBEC））；Barcelona Institute of Science and Technology (BIST)（巴塞罗那科学技术学院（BIST））；Universitat Pompeu Fabra (UPF)（庞培法布拉大学（UPF））；Catalan Institution for Research and Advanced Studies (ICREA)（加泰罗尼亚研究与高级研究机构（ICREA））
认知科学和人工智能的一个主要挑战是理解自主智能体如何在复杂的社会互动过程中获取和预测其他智能体的行为和精神状态。这样一个智能体如何对与之交互的其他智能体的目标、信念和行为进行建模？心理理论（Theory of Mind，ToM）的计算原理是什么？解决这些问题的深度学习方法无法更好地理解问题。在某种程度上，这是由于深度网络的黑箱性质，其中ToM的计算机制不容易揭示。在这里，我们考虑替代假设，试图模拟大脑如何实现ToM。特别是，我们提出了具体的和位于代理模型的基础上，分布式自适应控制理论来预测其他代理在五个不同的博弈论任务（和谐游戏，鹰鸽，鹿狩猎，囚徒困境和战斗的前任）的行动。我们的多层控制模型实现了从自适应到反应控制层的自上而下的预测和从反应到自适应层的自下而上的误差反馈。我们测试七种不同的代理模型（合作，贪婪，针锋相对，基于协商，理性，预测和其他的模型代理）之间的合作和竞争策略。我们发现，与纯粹的基于学习的策略相比，基于理性，预测和其他模型表型的概率学习代理在跨任务的博弈论指标中表现更好。我们的自主多代理模型捕获系统级的过程中的ToM和突出的体系结构原则的ToM从控制理论的角度来看。
