To address the intricate challenges of decentralized cooperative scheduling and motion planning in Autonomous Mobility-on-Demand (AMoD) systems, this paper introduces LMMCoDrive, a novel cooperative driving framework that leverages a Large Multimodal Model (LMM) to enhance traffic efficiency in dynamic urban environments. This framework seamlessly integrates scheduling and motion planning processes to ensure the effective operation of Cooperative Autonomous Vehicles (CAVs). The spatial relationship between CAVs and passenger requests is abstracted into a Bird's-Eye View (BEV) to fully exploit the potential of the LMM. Besides, trajectories are cautiously refined for each CAV while ensuring collision avoidance through safety constraints. A decentralized optimization strategy, facilitated by the Alternating Direction Method of Multipliers (ADMM) within the LMM framework, is proposed to drive the graph evolution of CAVs. Simulation results demonstrate the pivotal role and significant impact of LMM in optimizing CAV scheduling and enhancing decentralized cooperative optimization process for each vehicle. This marks a substantial stride towards achieving practical, efficient, and safe AMoD systems that are poised to revolutionize urban transportation. The code is available at https://github.com/henryhcliu/LMMCoDrive.
Alexa AI, Amazon Inc（亚马逊公司Alexa人工智能部门）；University of Southern California（南加州大学）
We propose a multimodal (vision-and-language) benchmark for cooperative and heterogeneous multi-agent learning. We introduce a benchmark multimodal dataset with tasks involving collaboration between multiple simulated heterogeneous robots in a rich multi-room home environment. We provide an integrated learning framework, multimodal implementations of state-of-the-art multi-agent reinforcement learning techniques, and a consistent evaluation protocol. Our experiments investigate the impact of different modalities on multi-agent learning performance. We also introduce a simple message passing me…
Real-time Vehicle-to-Vehicle Communication Based Network Cooperative Control System through Distributed Database and Multimodal Perception: Demonstrated in Crossroads
Zhejiang University（浙江大学）；University of Illinois Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）；University of Michigan（密歇根大学）
V2X-QA: A Comprehensive Reasoning Dataset and Benchmark for Multimodal Large Language Models in Autonomous Driving Across Ego, Infrastructure, and Cooperative Views
Department of Civil and Environmental Engineering, University of Wisconsin–Madison（威斯康星大学麦迪逊分校土木与环境工程系）；Department of Civil and Architectural Engineering and Construction Management, University of Wyoming（怀俄明大学土木与建筑工程及施工管理系）；School of Transportation, Southeast University（东南大学交通学院）
Department of Civil and Environmental Engineering, University of Wisconsin–Madison（威斯康星大学麦迪逊分校土木与环境工程系）；College of Computing and Data Science, Nanyang Technological University（南洋理工大学 computing and Data Science 学院）；College of Transportation, Tongji University（同济大学交通运输学院）
Cooperative table-carrying is a complex task due to the continuous nature of the action and state-spaces, multimodality of strategies, and the need for instantaneous adaptation to other agents. In this work, we present a method for predicting realistic motion plans for cooperative human-robot teams on the task. Using a Variational Recurrent Neural Network (VRNN) to model the variation in the trajectory of a human-robot team across time, we are able to capture the distribution over the team's future states while leveraging information from interaction history. The key to our approach is leverag…
V2X-WAM: A Cooperative World Action Model for End-to-End Autonomous Driving
ITS Center, Research Institute of Highway Ministry of Transport（交通运输部公路科学研究院智能交通系统中心）；State Key Lab of Intelligent Transportation System, Research Institute of Highway Ministry of Transport（交通运输部公路科学研究院智能交通系统国家重点实验室）；Department of Civil and Environmental Engineering, University of Wisconsin–Madison（威斯康星大学麦迪逊分校土木与环境工程系）；Intelligent Transportation Systems Research Center, Wuhan University of Technology（武汉理工大学智能交通系统研究中心）；Engineering Research Center of Transportation Information and Safety, Ministry of Education（教育部交通信息与安全工程研究中心）；School of Transportation, Inner Mongolia University（内蒙古大学交通学院）；College of Transportation, Tongji University（同济大学交通运输工程学院）
Technical University of Munich（慕尼黑工业大学）；Stanford University（斯坦福大学）；NVIDIA Research（英伟达研究院）
Autonomous systems and humans are increasingly sharing the same space. Robots work side by side or even hand in hand with humans to balance each other's limitations. Such cooperative interactions are ever more sophisticated. Thus, the ability to reason not just about a human's center of gravity position, but also its granular motion is an important prerequisite for human-robot interaction. Though, many algorithms ignore the multimodal nature of humans or neglect uncertainty in their motion forecasts. We present Motron, a multimodal, probabilistic, graph-structured model, that captures human's …
SCORP: Scene-Consistent Multi-agent Diffusion Planning with Stable Online Reinforcement Post-Training for Cooperative Driving
School of Information Science and Technology, Harbin Institute of Technology (Shenzhen)（哈尔滨工业大学（深圳）信息科学与技术学院）；Middle East Technology University (METU)（中东技术大学）；Robotics and Autonomous Systems Thrust, Hong Kong University of Science and Technology (Guangzhou)（香港科技大学（广州）机器人与自主系统方向）；School of Computer Science and Technology, Qinghai University（青海大学计算机科学与技术学院）
Edge-Based Multimodal Sensor Data Fusion with Vision Language Models (VLMs) for Real-time Autonomous Vehicle Accident Avoidance
Department of Civil & Environmental Engineering University of Utah（土木与环境工程系 犹他大学）；Zachry Department of Civil and Environmental Engineering Texas A&M University（扎克里系 土木与环境工程系 德克萨斯农工大学）；Department of Computer Science & Engineering Texas A&M University（计算机科学与工程系 德克萨斯农工大学）
