支持URLLC的车载网络中带包络更新的广义多目标强化学习
Generalized Multi-Objective Reinforcement Learning with Envelope Updates in URLLC-enabled Vehicular Networks
AI总结：
针对支持URLLC的多频段车载网络，提出带凸包络更新的广义MORL框架，联合优化网络选择与驾驶策略，可适配任意目标偏好，实现安全驾驶与通信性能的协同提升。
摘要：
我们提出了一种新型多目标强化学习（MORL，multi-objective reinforcement learning）框架，用于在工作于传统Sub-6GHz频段与太赫兹频率的多频段车载网络中，联合优化无线网络选择与自动驾驶策略。该框架的设计目标包括：1. 通过控制车辆的运动动力学（即速度与加速度），最大化交通流量并最小化碰撞；2. 增强超可靠低时延通信（URLLC，ultra-reliable low-latency communication）性能，同时最小化切换（HOs，handoffs）次数。我们将该问题建模为多目标马尔可夫决策过程（MOMDP，multi-objective Markov Decision Process），并针对冲突目标的偏好已知与未知两种场景分别设计了解决方案。具体而言，我们提出了一种新型包络MORL方法，能够在智能体未知目标偏好的情况下，生成适配多目标的策略。虽然该方法降低了对标量奖励的依赖，但策略有效性会随偏好不同而变化，这是一个亟待解决的挑战。为此，我们应用了广义版本的Bellman方程，通过优化多目标Q值的凸包络，学习统一的参数化表示，该表示能够生成适配所有可能偏好配置的最优策略。在初始学习阶段完成后，我们的智能体既可以在任意指定偏好下执行最优策略，也能够从少量数据样本中推断偏好。数值实验结果验证了基于包络的MORL方案的有效性，并揭示了车辆运动动力学、切换次数与通信数据速率之间相互依赖关系的有趣洞见。所提出的策略能够让自动驾驶汽车（AVs，autonomous vehicles）在提升连接性能的同时采取安全驾驶行为。
英文摘要：
We develop a novel multi-objective reinforcement learning (MORL) framework to jointly optimize wireless network selection and autonomous driving policies in a multi-band vehicular network operating on conventional sub-6GHz spectrum and Terahertz frequencies. The proposed framework is designed to 1. maximize the traffic flow and minimize collisions by controlling the vehicle's motion dynamics (i.e., speed and acceleration), and 2. enhance the ultra-reliable low-latency communication (URLLC) while minimizing handoffs (HOs). We cast this problem as a multi-objective Markov Decision Process (MOMDP) and develop solutions for both predefined and unknown preferences of the conflicting objectives. Specifically, we develop a novel envelope MORL solution which develops policies that address multiple objectives with unknown preferences to the agent. While this approach reduces reliance on scalar rewards, policy effectiveness varying with different preferences is a challenge. To address this, we apply a generalized version of the Bellman equation and optimize the convex envelope of multi-objective Q values to learn a unified parametric representation capable of generating optimal policies across all possible preference configurations. Following an initial learning phase, our agent can execute optimal policies under any specified preference or infer preferences from minimal data samples. Numerical results validate the efficacy of the envelope-based MORL solution and demonstrate interesting insights related to the inter-dependency of vehicle motion dynamics, HOs, and the communication data rate. The proposed policies enable autonomous vehicles (AVs) to adopt safe driving behaviors with improved connectivity.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
