Millimeter-wave (mmWave) base station can offer abundant high capacity channel resources toward connected vehicles so that quality-of-service (QoS) of them in terms of downlink throughput can be highly improved. The mmWave base station can operate among existing base stations (e.g., macro-cell base station) on non-overlapped channels among them and the vehicles can make decision what base station to associate, and what channel to utilize on heterogeneous networks. Furthermore, because of the non-omni property of mmWave communication, the vehicles decide how to align the beam direction toward mmWave base station to associate with it. However, such joint problem requires high computational cost, which is NP-hard and has combinatorial features. In this paper, we solve the problem in 3-tier heterogeneous vehicular network (HetVNet) with multi-agent deep reinforcement learning (DRL) in a way that maximizes expected total reward (i.e., downlink throughput) of vehicles. The multi-agent deep deterministic policy gradient (MADDPG) approach is introduced to achieve optimal policy in continuous action domain.
Amazon（亚马逊）；University of Massachusetts Lowell（马萨诸塞大学洛厄尔分校）
Given the aging infrastructure and the anticipated growing number of highway work zones in the United States, it is important to investigate work zone merge control, which is critical for improving work zone safety and capacity. This paper proposes and evaluates a novel highway work zone merge control strategy based on cooperative driving behavior enabled by artificial intelligence. The proposed method assumes that all vehicles are fully automated, connected and cooperative. It inserts two metering zones in the open lane to make space for merging vehicles in the closed lane. In addition, each …
Carnegie Mellon University（卡内基梅隆大学）；InfoTech Labs, Toyota Motor North America R&D（丰田北美研发公司InfoTech实验室）
Sensor-based perception on vehicles are becoming prevalent and important to enhance the road safety. Autonomous driving systems use cameras, LiDAR, and radar to detect surrounding objects, while human-driven vehicles use them to assist the driver. However, the environmental perception by individual vehicles has the limitations on coverage and/or detection accuracy. For example, a vehicle cannot detect objects occluded by other moving/static obstacles. In this paper, we present a cooperative perception scheme with deep reinforcement learning to enhance the detection accuracy for the surrounding…
University of Oulu（奥卢大学）；University of the Basque Country UPV/EHU（巴斯克大学）；Virginia Tech（弗吉尼亚理工大学）
Cooperative perception plays a vital role in extending a vehicle's sensing range beyond its line-of-sight. However, exchanging raw sensory data under limited communication resources is infeasible. Towards enabling an efficient cooperative perception, vehicles need to address the following fundamental question: What sensory data needs to be shared?, at which resolution?, and with which vehicles? To answer this question, in this paper, a novel framework is proposed to allow reinforcement learning (RL)-based vehicular association, resource block (RB) allocation, and content selection of cooperati…
Reinforcement learning in cooperative multi-agent settings has recently advanced significantly in its scope, with applications in cooperative estimation for advertising, dynamic treatment regimes, distributed control, and federated learning. In this paper, we discuss the problem of cooperative multi-agent RL with function approximation, where a group of agents communicates with each other to jointly solve an episodic MDP. We demonstrate that via careful message-passing and cooperative value iteration, it is possible to achieve near-optimal no-regret learning even with a fixed constant communic…
Karlsruhe Institute of Technology（卡尔斯鲁厄理工学院）；Robert Bosch GmbH（罗伯特·博世有限公司）
Cooperative trajectory planning methods for automated vehicles can solve traffic scenarios that require a high degree of cooperation between traffic participants. However, for cooperative systems to integrate into human-centered traffic, the automated systems must behave human-like so that humans can anticipate the system's decisions. While Reinforcement Learning has made remarkable progress in solving the decision-making part, it is non-trivial to parameterize a reward model that yields predictable actions. This work employs feature-based Maximum Entropy Inverse Reinforcement Learning combine…
Queensland University of Technology（昆士兰科技大学）；Queensland Centre of Robotics（昆士兰机器人中心）
Many hierarchical reinforcement learning algorithms utilise a series of independent skills as a basis to solve tasks at a higher level of reasoning. These algorithms don't consider the value of using skills that are cooperative instead of independent. This paper proposes the Cooperative Consecutive Policies (CCP) method of enabling consecutive agents to cooperatively solve long time horizon multi-stage tasks. This method is achieved by modifying the policy of each agent to maximise both the current and next agent's critic. Cooperatively maximising critics allows each agent to take actions that…
In this paper, we study how to learn an appropriate lane changing strategy for autonomous vehicles by using deep reinforcement learning. We show that the reward of the system should consider the overall traffic efficiency instead of the travel efficiency of an individual vehicle. In summary, cooperation leads to a more harmonic and efficient traffic system rather than competition
Beijing National Research Center for Information Science and Technology（北京信息科学与技术国家研究中心）；Tsinghua University（清华大学）
Cooperative perception of connected vehicles comes to the rescue when the field of view restricts stand-alone intelligence. While raw-level cooperative perception preserves most information to guarantee accuracy, it is demanding in communication bandwidth and computation power. Therefore, it is important to schedule the most beneficial vehicle to share its sensor in terms of supplementary view and stable network connection. In this paper, we present a model of raw-level cooperative perception and formulate the energy minimization problem of sensor sharing scheduling as a variant of the Multi-A…
Alexa AI, Amazon Inc（亚马逊公司Alexa人工智能部门）；University of Southern California（南加州大学）
We propose a multimodal (vision-and-language) benchmark for cooperative and heterogeneous multi-agent learning. We introduce a benchmark multimodal dataset with tasks involving collaboration between multiple simulated heterogeneous robots in a rich multi-room home environment. We provide an integrated learning framework, multimodal implementations of state-of-the-art multi-agent reinforcement learning techniques, and a consistent evaluation protocol. Our experiments investigate the impact of different modalities on multi-agent learning performance. We also introduce a simple message passing me…
We present a new model-based reinforcement learning algorithm, Cooperative Prioritized Sweeping, for efficient learning in multi-agent Markov decision processes. The algorithm allows for sample-efficient learning on large problems by exploiting a factorization to approximate the value function. Our approach only requires knowledge about the structure of the problem in the form of a dynamic decision network. Using this information, our method learns a model of the environment and performs temporal difference updates which affect multiple joint states and actions at once. Batch updates are addit…
University of Illinois at Urbana-Champaign (UIUC)（伊利诺伊大学厄巴纳-香槟分校）；École Polytechnique Fédérale de Lausanne (EPFL)（洛桑联邦理工学院）
This article reviews recent advances in multi-agent reinforcement learning algorithms for large-scale control systems and communication networks, which learn to communicate and cooperate. We provide an overview of this emerging field, with an emphasis on the decentralized setting under different coordination protocols. We highlight the evolution of reinforcement learning algorithms from single-agent to multi-agent systems, from a distributed optimization perspective, and conclude with future directions and challenges, in the hope to catalyze the growing synergy among distributed optimization, …
This paper studies the cooperative source seeking problem via a networked multi-vehicle system. In contrast to existing literature, the multi-vehicle system is controlled to the source position that maximizes aggregated multiple unknown scalar fields and each sensor-enabled vehicle only samples measurements of one scalar field. Thus, a single vehicle is unable to localize the source and has to cooperate with its neighboring vehicles. By jointly exploiting the ideas of the consensus algorithm and the stochastic extremum seeking (ES), this paper proposes novel distributed stochastic ES controlle…
Cooperative Localization is expected to play a crucial role in various applications in the field of Connected and Autonomous vehicles (CAVs). Future 5G wireless systems are expected to enable cost-effective Vehicle-to-Everything (V2X)systems, allowing CAVs to share with the other entities of the network the data they collect and measure. Typical measurement models usually deployed for this problem, are absolute position from Global Positioning System (GPS), relative distance and azimuth angle to neighbouring vehicles, extracted from Light Detection and Ranging (LIDAR) or Radio Detection and Ra…
Vehicle positioning is considered a key element in autonomous driving systems. While conventional positioning requires the use of GPS and/or beacon signals from network infrastructure for triangulation, they are sensitive to multi-path and signal obstruction. However, recent proposals like the Channel-SLAM method showed it was possible in principle to in fact leverage multi-path to improve positioning of a single vehicle. In this paper, we derive a cooperative Channel-SLAM framework, which is referred as Team Channel-SLAM. Different from the previous work, Team Channel-SLAM not only exploits t…
School of Mechanical Engineering, Beijing Institute of Technology（北京理工大学机械工程学院）
This paper develops a stop line aided cooperative positioning framework for connected vehicles, which creatively utilizes the location of the stop-line to achieve the positioning enhancement for a vehicular ad-hoc network (VANET) in intersection scenarios via Vehicle-to-Vehicle (V2V) communication. Firstly, a self-positioning correction scheme for the first stopped vehicle is presented, which applied the stop line information as benchmarks to correct the GNSS/INS positioning results. Then, the local observations of each vehicle are fused with the position estimates of other vehicles and the in…
High-definition (HD) map transmission is considered as a key technology for automatic driving, which enables vehicles to obtain the precise road and surrounding environment information for further localization and navigation. Guaranteeing the huge requirement of HD map data, the objective of this paper is to reduce the power consumption of vehicular networks. By leveraging the mobile rule of vehicles, a collaborative vehicle to everything (V2X) transmission scheme is proposed for the HD map transmission. Numerical results indicate that the proposed scheme can satisfy the transmission rate requ…
This paper addresses vehicle positioning, a topic whose importance has risen dramatically in the context of future autonomous driving systems. While classical methods that use GPS and/or beacon signals from network infrastructure for triangulation tend to be sensitive to multi-paths and signal obstruction, our method exhibits robustness with respect to such phenomena. Our approach builds on the recently proposed Channel-SLAM method which first enabled leveraging of multi-path so as to improve (single) vehicle positioning. Here, we propose a cooperative mapping approach which builds upon the Ch…
Using the multiple-model (MM) probability hypothesis density (PHD) filter, millimeter wave (mmWave) radio simultaneous localization and mapping (SLAM) in vehicular scenarios is susceptible to movements of objects, in particular vehicles driving in parallel with the ego vehicle. We propose and evaluate two countermeasures to track vehicle scatterers (VSs) in mmWave radio MM-PHD-SLAM. First, locally at each vehicle, we generate and treat the VS map PHD in the context of Bayesian recursion, and modify vehicle state correction with the VS map PHD. Second, in the global map fusion process at the ba…
Internet of Vehicles (IoV) is expected to become the central infrastructure to provide advanced services to connected vehicles and users for higher transportation efficiency and security. A variety of emerging applications/services bring explosively growing demands for mobile data traffic between connected vehicles and roadside units (RSU), imposing the significant challenge of spectrum scarcity to IoV. In this paper, we propose a cooperative semantic-aware architecture to convey essential semantics from collaborated users to servers for lowering the data traffic. In contrast to current soluti…
University of Electronic Science and Technology of China（电子科技大学）；University of Essex（埃塞克斯大学）
Research on the sixth generation cellular networks (6G) is gaining huge momentum to achieve ubiquitous wireless connectivity. Connected autonomous driving (CAV) is a critical vertical envisioned for 6G, holding great potentials of improving road safety, road and energy efficiency. However the stringent service requirements of CAV applications on reliability, latency and high speed communications will present big challenges to 6G networks. New channel access algorithms and intelligent control schemes for connected vehicles are needed for 6G supported CAV. In this paper, we investigated 6G suppo…
Cooperative overtaking is believed to have the capability of improving road safety and traffic efficiency by means of the real-time information exchange between traffic participants, including road infrastructures, nearby vehicles and others. In this paper, we focused on the critical issues of modeling, computation, and analysis of cooperative overtaking and made it playing a key role in the road overtaking area. In detail, for the purpose of extending the awareness of the surrounding environment, the lane markings in front of ego vehicle were detected and modeled with Bezier curve using an on…
Short-term future of automated driving can be imagined as a hybrid scenario in which both automated and human-driven vehicles co-exist in the same environment. In order to address the needs of such road configuration, many technology solutions such as vehicular communication and predictive control for automated vehicles have been introduced in the literature. Both aforementioned solutions rely on driving data of the human driver. In this work, we investigate the currently available driving datasets and introduce a real-world maneuver-based driving dataset that is collected during our urban dri…
This paper offers a technique for estimating collision risk for automated ground vehicles engaged in cooperative sensing. The technique allows quantification of (i) risk reduced due to cooperation, and (ii) the increased accuracy of risk assessment due to cooperation. If either is significant, cooperation can be viewed as a desirable practice for meeting the stringent risk budget of increasingly automated vehicles; if not, then cooperation - with its various drawbacks - need not be pursued. Collision risk is evaluated over an ego vehicle's trajectory based on a dynamic probabilistic occupancy …
The application of cooperative localization in vehicular networks is attractive to improve accuracy and coverage. Conventional distance measurements between vehicles are limited by the need for synchronization and provide no heading information of the vehicle. To address this, we present a cooperative localization algorithm using posterior linearization belief propagation (PLBP) utilizing angle-of-arrival (AoA)-only measurements. Simulation results show that both directional and positional root mean squared error (RMSE) of vehicles can be decreased significantly and converge to a low value in …
Cooperative vehicular networks will play a vital role in the coming years to implement various intelligent transportation-related applications. Both vehicle-to-vehicle (V2V) and vehicle-to-infrastructure (V2I) communications will be needed to reliably disseminate information in a vehicular network. In this regard, a roadside unit (RSU) equipped with multiple antennas can improve the network capacity. While the traditional approaches assume antennas to experience independent fading, we consider a more practical uplink scenario where antennas at the RSU experience correlated fading. In particula…
