Challenges in real-world robotic applications often stem from managing multiple, dynamically varying entities such as neighboring robots, manipulable objects, and navigation goals. Existing multi-agent control strategies face scalability limitations, struggling to handle arbitrary numbers of entities. Additionally, they often rely on engineered heuristics for assigning entities among agents. We propose a data driven approach to address these limitations by introducing a decentralized control system using neural network policies trained in simulation. Leveraging permutation invariant neural network architectures and model-free reinforcement learning, our approach allows control agents to autonomously determine the relative importance of different entities without being biased by ordering or limited by a fixed capacity. We validate our approach through both simulations and real-world experiments involving multiple wheeled-legged quadrupedal robots, demonstrating their collaborative control capabilities. We prove the effectiveness of our architectural choice through experiments with three exemplary multi-entity problems. Our analysis underscores the pivotal role of the end-to-end trained permutation invariant encoders in achieving scalability and improving the task performance in multi-object manipulation or multi-goal navigation problems. The adaptability of our policy is further evidenced by its ability to manage varying numbers of entities in a zero-shot manner, showcasing near-optimal autonomous task distribution and collision avoidance behaviors.
T-Rex: Task-Adaptive Spatial Representation Extraction for Robotic Manipulation with Vision-Language Models
School of Software Engineering, South China University of Technology（软件工程学院，华南理工大学）；School of Future Technology, South China University of Technology（未来技术学院，华南理工大学）；Shien-Ming Wu School of Intelligent Engineering, South China University of Technology（智能工程学院，华南理工大学）
JAIST（北陆先端科学技术大学院大学）；Kyushu University（九州大学）；Aix-Marseille University（艾克斯-马赛大学）；CNRS（法国国家科学研究中心）；University of Rome “Sapienza”（罗马第一大学）；University of Ottawa（渥太华大学）；Carleton University（卡尔顿大学）
Factored World Models for Zero-Shot Generalization in Robotic Manipulation
Khoury College of Computer Sciences, Northeastern University（东北大学库里计算机科学学院）；Informatics Institute, University of Amsterdam（阿姆斯特丹大学信息学研究所）；Google Research, Brain Team（谷歌研究院大脑团队）
Vislab, Institute for Systems and Robotics—Lisboa, Instituto Superior Técnico, Universidade de Lisboa, Portugal（葡萄牙里斯本系统与机器人研究所Vislab，理工学院里斯本高等技术大学，里斯本大学）；Department of Marine, Electrical, Electronic and Telecommunications Engineering, University of Genoa, Italy（意大利热那亚大学海洋、电气、电子与电信工程系）；Pattern Analysis & Computer Vision (PAVIS), Istituto Italiano di Tecnologia (IIT), Genoa, Italy（意大利热那亚理工学院模式分析与计算机视觉（PAVIS））；Deep Visual Learning (DVL), Fondazione Bruno Kessler, Trento, Italy（意大利特伦托布鲁诺·凯瑟基金会深度视觉学习（DVL））
Synthetic Data Generation and Vision-based Wrinkle and Keypoint Detection for Bimanual Cloth Manipulation
Department of Engineering, University of Luxembourg（卢森堡大学工程系）；School of Electrical and Electronic Engineering, Nanyang Technological University（南洋理工大学电子与电气工程学院）；Université de Lorraine, Arts et Metiers Institute of Technology, LCFC（洛林大学，艺术与工艺技术学院，LCFC）
