Incomplete knowledge of the environment leads an agent to make decisions under uncertainty. One of the major dilemmas in Reinforcement Learning (RL) where an autonomous agent has to balance two contrasting needs in making its decisions is: exploiting the current knowledge of the environment to maximize the cumulative reward as well as exploring actions that allow improving the knowledge of the environment, hopefully leading to higher reward values (exploration-exploitation trade-off). Concurrently, another relevant issue regards the full observability of the states, which may not be assumed in all applications. For instance, when 2D images are considered as input in an RL approach used for finding the best actions within a 3D simulation environment. In this work, we address these issues by deploying and testing several techniques to balance exploration and exploitation trade-off on partially observable systems for predicting steering wheels in autonomous driving scenarios. More precisely, the final aim is to investigate the effects of using both adaptive and deterministic exploration strategies coupled with a Deep Recurrent Q-Network. Additionally, we adapted and evaluated the impact of a modified quadratic loss function to improve the learning phase of the underlying Convolutional Recurrent Neural Network. We show that adaptive methods better approximate the trade-off between exploration and exploitation and, in general, Softmax and Max-Boltzmann strategies outperform epsilon-greedy techniques.
Balancing Exploration and Exploitation in Hierarchical Reinforcement Learning via Latent Landmark Graphs
Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；School of Future Technology, University of Chinese Academy of Sciences（中国科学院大学未来技术学院）；School of Artificial Intelligence, University of Chinese Academy of Sciences（中国科学院大学人工智能学院）
Semantic-Space Exploration and Exploitation in RLVR for LLM Reasoning
Tsinghua Shenzhen International Graduate School, Tsinghua University（清华大学深圳国际研究生院，清华大学）；University of California, Los Angeles（加州大学洛杉矶分校）；Shenzhen Technology University（深圳技术大学）
Tiered Reinforcement Learning: Pessimism in the Face of Uncertainty and Constant Regret
University of Illinois at Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）；Microsoft Research Asia（微软亚洲研究院）
我们提出了一个新的学习框架，捕捉分层结构的许多现实世界中的用户交互应用程序，用户可以分为两组，根据他们不同的容忍度的探索风险，并应单独对待。在此设置中，我们同时维护两个策略$π^{\text{O}}$和$π^{\text{E}}$：$π^{\text{O}}$（“O”表示“在线”）与第一层中风险容忍度更高的用户交互，并通过像往常一样平衡探索和利用来最大限度地减少遗憾，而$π^{\text{E}}$（“E”表示“利用”）则专门关注利用到目前为止收集的数据对第二层中风险规避用户的利用。一个重要的问题是这样的分离是否比标准在线设置（即，$π^{\text{E}}=π^{\text{O}}$）。我们单独考虑间隙独立与~间隙相关设置。对于前者，我们证明了从极大极小的角度来看，分离确实是没有好处的。对于后者，我们表明，如果选择Pessibility Value Iteration作为开发算法来产生$π^{\text{E}}$，我们可以为风险厌恶的用户实现独立于情节$K$数量的恒定遗憾，这与相同设置中任何在线RL算法的$Ω(\log K)$遗憾形成鲜明对比，而$π^{\text{O}}$的后悔（几乎）保持其在线后悔最优性，并且不需要为了$π^{\text{E}}$的成功而妥协。
DGRO: Enhancing LLM Reasoning via Exploration-Exploitation Control and Reward Variance Management
School of Mathematics and Statistics（数学与统计学学院）；Beijing Jiaotong University（北京交通大学）；Yau Mathematical Sciences Center（叶中科学中心）；Tsinghua University（清华大学）；Department of Mathematical Sciences（数学科学系）；Beijing, China（中国北京）；Zhongguancun Academy（中关村学院）；Yanqi Lake Beijing Institute of Mathematical Sciences and Applications（燕琪湖数学科学与应用研究所）；Academy of Mathematics and Systems Science（数学与系统科学研究院）；Chinese Academy of Sciences（中国科学院）；Microsoft Research AI4Science（微软研究院AI4Science）
All Models are Wrong, Knowing Where is Useful: On Model Uncertainty in Reinforcement Learning
German Federal Ministry of Research, Technology and Space (BMFTR)（德国联邦研究、技术和空间部）；Robotics Institute Germany (RIG)（德国机器人研究所）；Institute for Data Science in Mechanical Engineering, RWTH Aachen University（机械工程数据科学研究所，亚琛工业大学）；NHR Center NHR4CES at RWTH Aachen University（亚琛工业大学NHR4CES中心）
Institute of Systems Science, AMSS, Chinese Academy of Sciences（中国科学院数学与系统科学研究院系统科学研究所）；Nanjing University（南京大学）；Central South University（中南大学）；City University of Hong Kong（香港城市大学）；Washington University in St. Louis（圣路易斯华盛顿大学）
