Energy management strategies (EMSs) are the most significant components in hybrid electric vehicles (HEVs) because they decide the potential of energy conservation and emission reduction. This work presents a transferred EMS for a parallel HEV via combining the reinforcement learning method and driving conditions recognition. First, the Markov decision process (MDP) and the transition probability matrix are utilized to differentiate the driving conditions. Then, reinforcement learning algorithms are formulated to achieve power split controls, in which Q-tables are tuned by current driving situations. Finally, the proposed transferred framework is estimated and validated in a parallel hybrid topology. Its advantages in computational efficiency and fuel economy are summarized and proved.
School of Robot Engineering, Yangtze Normal University（长江师范学院机器人工程学院）；College of Automotive Engineering, Chongqing University（重庆大学汽车工程学院）；School of intelligent manufacturing engineering, Chongqing University of Arts and Sciences（重庆文理学院智能制造工程学院）；Department of Mechanical and Mechatronics Engineering, University of Waterloo（滑铁卢大学机械与机电工程学院）
This paper proposes an adaptive energy management strategy for hybrid electric vehicles by combining deep reinforcement learning (DRL) and transfer learning (TL). This work aims to address the defect of DRL in tedious training time. First, an optimization control modeling of a hybrid tracked vehicle is built, wherein the elaborate powertrain components are introduced. Then, a bi-level control framework is constructed to derive the energy management strategies (EMSs). The upper-level is applying the particular deep deterministic policy gradient (DDPG) algorithms for EMS training at different sp…
Reinforcement learning-based (RL-based) energy management strategy (EMS) is considered a promising solution for the energy management of electric vehicles with multiple power sources. It has been shown to outperform conventional methods in energy management problems regarding energy-saving and real-time performance. However, previous studies have not systematically examined the essential elements of RL-based EMS. This paper presents an empirical analysis of RL-based EMS in a Plug-in Hybrid Electric Vehicle (PHEV) and Fuel Cell Electric Vehicle (FCEV). The empirical analysis is developed in fou…
Shared Automated Vehicles (SAVs) Fleets companies are starting pilot projects nationwide. In 2020 in Fairfax Virginia it was announced the first Shared Autonomous Vehicle Fleet pilot project in Virginia. SAVs promise to improve quality of life. However, SAVs will also induce some negative externalities by generating excessive vehicle miles traveled (VMT), which leads to more congestions, energy consumption, and emissions. The excessive VMT are primarily generated via empty relocation process. Reinforcement Learning based algorithms are being researched as a possible solution to solve some of t…
Pakistan Institute of Engineering and Applied Sciences（巴基斯坦工程与应用科学学院）；Carleton University（卡尔顿大学）
Deep Reinforcement Learning (DRL) uses diverse, unstructured data and makes RL capable of learning complex policies in high dimensional environments. Intelligent Transportation System (ITS) based on Autonomous Vehicles (AVs) offers an excellent playground for policy-based DRL. Deep learning architectures solve computational challenges of traditional algorithms while helping in real-world adoption and deployment of AVs. One of the main challenges in AVs implementation is that it can worsen traffic congestion on roads if not reliably and efficiently managed. Considering each vehicle's holistic e…
University of Birmingham（伯明翰大学）；Zhejiang University（浙江大学）
The future transportation system will be a multi-agent network where connected AI agents can work together to address the grand challenges in our age, e.g., mitigation of real-world driving energy consumption. Distinguished from the existing research on vehicle energy management, which decoupled multiple inputs and multiple outputs (MIMO) control into single-output(MISO) control, this paper studied a multi-agent deep reinforcement learning (MADRL) framework to deal with multiple control outputs simultaneously. A new hand-shaking strategy is proposed for the DRL agents by introducing an indepen…
Modeling difficulty, time-varying model, and uncertain external inputs are the main challenges for energy management of fuel cell hybrid electric vehicles. In the paper, a fuzzy reinforcement learning-based energy management strategy for fuel cell hybrid electric vehicles is proposed to reduce fuel consumption, maintain the batteries' long-term operation, and extend the lifetime of the fuel cells system. Fuzzy Q-learning is a model-free reinforcement learning that can learn itself by interacting with the environment, so there is no need for modeling the fuel cells system. In addition, frequent…
Connected Autonomous Vehicles will make autonomous intersection management a reality replacing traditional traffic signal control. Autonomous intersection management requires time and speed adjustment of vehicles arriving at an intersection for collision-free passing through the intersection. Due to its computational complexity, this problem has been studied only when vehicle arrival times towards the vicinity of the intersection are known beforehand, which limits the applicability of these solutions for real-time deployment. To solve the real-time autonomous traffic intersection management pr…
Adversarial examples are firstly investigated in the area of computer vision: by adding some carefully designed ''noise'' to the original input image, the perturbed image that cannot be distinguished from the original one by human, can fool a well-trained classifier easily. In recent years, researchers also demonstrated that adversarial examples can mislead deep reinforcement learning (DRL) agents on playing video games using image inputs with similar methods. However, although DRL has been more and more popular in the area of intelligent transportation systems, there is little research invest…
Beijing Institute of Technology（北京理工大学）；Delft University of Technology（代尔夫特理工大学）
The deep reinforcement learning-based energy management strategies (EMS) have become a promising solution for hybrid electric vehicles (HEVs). When driving cycles are changed, the neural network will be retrained, which is a time-consuming and laborious task. A more efficient way of choosing EMS is to combine deep reinforcement learning (DRL) with transfer learning, which can transfer knowledge of one domain to the other new domain, making the network of the new domain reach convergence values quickly. Different exploration methods of DRL, including adding action space noise and parameter spac…
This paper studies the problem of spectrum shortage in an unmanned aerial vehicle (UAV) network during critical missions such as wildfire monitoring, search and rescue, and disaster monitoring. Such applications involve a high demand for high-throughput data transmissions such as real-time video-, image-, and voice- streaming where the assigned spectrum to the UAV network may not be adequate to provide the desired Quality of Service (QoS). In these scenarios, the aerial network can borrow an additional spectrum from the available terrestrial networks in the trade of a relaying service for them…
Blockchain-based Pseudonym Management for Vehicle Twin Migrations in Vehicular Edge Metaverse
School of Automation, Guangdong University of Technology（广东工业大学自动化学院）；School of Computer Science and Engineering, Nanyang Technological University（南洋理工大学计算机科学与工程学院）；School of Electronic Science and Engineering, Nanjing University（南京大学电子科学与工程学院）；Department of Electrical and Computer Engineering, Auburn University（奥本大学电气与计算机工程系）
Skalar Systems GmbH（斯卡拉系统有限责任公司）；FH JOANNEUM – University of Applied Sciences（约阿内高等专业学院）；Know-Center GmbH（诺中心有限责任公司）；Graz University of Technology（格拉茨技术大学）；Washington University in St. Louis（圣路易斯华盛顿大学）
We investigate the feasibility of deploying Deep-Q based deep reinforcement learning agents to job-shop scheduling problems in the context of modular production facilities, using discrete event simulations for the environment. These environments are comprised of a source and sink for the parts to be processed, as well as (several) workstations. The agents are trained to schedule automated guided vehicles to transport the parts back and forth between those stations in an optimal fashion. Starting from a very simplistic setup, we increase the complexity of the environment and compare the agents'…
Chongqing University（重庆大学）；University of Waterloo（滑铁卢大学）
Knowledge transfer is a promising concept to achieve real-time decision-making for autonomous vehicles. This paper constructs a transfer deep reinforcement learning framework to transform the driving tasks in inter-section environments. The driving missions at the un-signalized intersection are cast into a left turn, right turn, and running straight for automated vehicles. The goal of the autonomous ego vehicle (AEV) is to drive through the intersection situation efficiently and safely. This objective promotes the studied vehicle to increase its speed and avoid crashing other vehicles. The dec…
Increasing the fuel economy of hybrid electric vehicles (HEVs) and extended range electric vehicles (EREVs) through optimization-based energy management strategies (EMS) has been an active research area in transportation. However, it is difficult to apply optimization-based EMS to current in-use EREVs because insufficient knowledge is known about future trips, and because such methods are computationally expensive for large-scale deployment. As a result, most past research has been validated on standard driving cycles or on recorded high-resolution data from past real driving cycles. This pape…
Chongqing University（重庆大学）；Chongqing Normal University（重庆师范大学）；Chongqing University of Arts and Sciences（重庆文理学院）
Real-time applications of energy management strategies (EMSs) in hybrid electric vehicles (HEVs) are the harshest requirements for researchers and engineers. Inspired by the excellent problem-solving capabilities of deep reinforcement learning (DRL), this paper proposes a real-time EMS via incorporating the DRL method and transfer learning (TL). The related EMSs are derived from and evaluated on the real-world collected driving cycle dataset from Transportation Secure Data Center (TSDC). The concrete DRL algorithm is proximal policy optimization (PPO) belonging to the policy gradient (PG) tech…
The connected vehicle paradigm empowers vehicles with the capability to communicate with neighboring vehicles and infrastructure, shifting the role of vehicles from a transportation tool to an intelligent service platform. Meanwhile, the transportation electrification pushes forward the electric vehicle (EV) commercialization to reduce the greenhouse gas emission by petroleum combustion. The unstoppable trends of connected vehicle and EVs transform the traditional vehicular system to an electric vehicular network (EVN), a clean, mobile, and safe system. However, due to the mobility and heterog…
Motivated by the concerns on transported fuel consumption and global air pollution, industrial engineers, and academic researchers have made many efforts to construct more efficient and environment-friendly vehicles. Hybrid electric vehicles (HEVs) are the representative ones because they can satisfy the power demand by coordinating energy supplements among different energy storage devices. To achieve this goal, energy management approaches are crucial technology, and driving cycles are the critical influence factor. Therefore, this paper aims to summarize driving cycle-driven energy managemen…
The charging load from Electric vehicles (EVs) is modeled as deferrable load, meaning that the power consumption can be shifted to different time windows to achieve various grid objectives. In local community scenarios, EVs are considered as controllable storage devices in a global optimization problem together with other microgrid components, such as the building load, renewable generations, and battery energy storage system, etc. However, the uncertainties in the driver behaviors have tremendous impact on the cost effectiveness of microgrid operations, which has not been fully explored in pr…
Despite the significant advances in vehicle automation and electrification, the next-decade aspirations for massive deployments of autonomous electric mobility on demand (AEMoD) services are still threatened by two major bottlenecks, namely the computational and charging delays. This paper proposes a solution for these two challenges by suggesting the use of fog computing for AEMoD systems, and developing an optimized charging scheme for its vehicles with and multi-class dispatching scheme for the customers. A queuing model representing the proposed multi-class management scheme with sub-class…
AoI-Driven Queue Management and Power Control in V2V Networks: A GNN-Enhanced MARL Approach
The National Mobile Communications Research Laboratory, Southeast University（东南大学）；College of Electronics and Information Engineering, Shenzhen University（深圳大学电子与信息工程学院）
Millimeter-wave (mmwave) communication represents a potential solution to capacity shortage in vehicular networks. However, effective beam alignment between senders and receivers requires accurate knowledge of the vehicles' position for fast beam steering, which is often impractical to obtain in real time. We address this problem by leveraging the traffic signals regulating vehicular mobility: as an example, we may coordinate beams with red traffic lights, as they correspond to higher vehicle densities and lower speeds. To evaluate our intuition, we propose a tractable, yet accurate, mmwave co…
