Uncertainty-Aware Rank-One MIMO Q Network Framework for Accelerated Offline Reinforcement Learning
School of Electrical Engineering, Korea Advanced Institute of Science and Technology（韩国科学技术院电子工程学院）；Department of Artificial Intelligence, Korea University（韩国大学人工智能系）
Thanh Nguyen, Tung Luu, Tri Ton, Sungwoong Kim, Chang D. Yoo
Offline reinforcement learning (RL) has garnered significant interest due to its safe and easily scalable paradigm. However, training under this paradigm presents its own challenge: the extrapolation error stemming from out-of-distribution (OOD) data. Existing methodologies have endeavored to address this issue through means like penalizing OOD Q-values or imposing similarity constraints on the learned policy and the behavior policy. Nonetheless, these approaches are often beset by limitations such as being overly conservative in utilizing OOD data, imprecise OOD data characterization, and significant computational overhead. To address these challenges, this paper introduces an Uncertainty-Aware Rank-One Multi-Input Multi-Output (MIMO) Q Network framework. The framework aims to enhance Offline Reinforcement Learning by fully leveraging the potential of OOD data while still ensuring efficiency in the learning process. Specifically, the framework quantifies data uncertainty and harnesses it in the training losses, aiming to train a policy that maximizes the lower confidence bound of the corresponding Q-function. Furthermore, a Rank-One MIMO architecture is introduced to model the uncertainty-aware Q-function, \TP{offering the same ability for uncertainty quantification as an ensemble of networks but with a cost nearly equivalent to that of a single network}. Consequently, this framework strikes a harmonious balance between precision, speed, and memory efficiency, culminating in improved overall performance. Extensive experimentation on the D4RL benchmark demonstrates that the framework attains state-of-the-art performance while remaining computationally efficient. By incorporating the concept of uncertainty quantification, our framework offers a promising avenue to alleviate extrapolation errors and enhance the efficiency of offline RL.
Efficient Beam Selection for ISAC in Cell-Free Massive MIMO via Digital Twin-Assisted Deep Reinforcement Learning
National Mobile Communications Research Laboratory, Frontiers Science Center for Mobile Information Communication and Security, Southeast University（国家移动通信研究中心，移动信息通信与安全前沿科学中心，东南大学）；Pervasive Communications Center, Purple Mountain Laboratories（普适通信中心，紫金山实验室）；School of Electronic and Optical Engineering, Nanjing University of Science and Technology（电子与光学工程学院，南京理工大学）
SHARP: Sleep-based Hierarchical Accelerated Replay for Long Range Non-Stationary Temporal Pattern Recognition
Department of Computer Engineering, University of Texas at San Antonio, USA（德克萨斯大学圣安东尼奥分校计算机工程系）；Department of Computer Science, University of Rochester, USA（罗切斯特大学计算机科学系）；Department of Psychology, University of Texas at San Antonio, USA（德克萨斯大学圣安东尼奥分校心理学系）
