Pretrained foundation models have exhibited extraordinary in-context learning performance, allowing zero-shot generalization to new tasks not encountered during pretraining. In the case of reinforcement learning (RL), in-context RL (ICRL) emerges when pretraining FMs on decision-making problems in an autoregressive-supervised manner. Nevertheless, current state-of-the-art ICRL algorithms, like Algorithm Distillation, Decision Pretrained Transformer and Decision Importance Transformer, impose stringent requirements on the pretraining dataset concerning the source policies, context information, and action labels. Notably, these algorithms either demand optimal policies or require varying degrees of well-trained behavior policies for all pretraining environments. This significantly hinders the application of ICRL to real-world scenarios, where acquiring optimal or well-trained policies for a substantial volume of real-world training environments can be intractable. To overcome this challenge, we introduce a novel approach, termed State-Action Distillation (SAD), that allows to generate an effective pretraining dataset guided solely by random policies. In particular, SAD selects query states and corresponding action labels by distilling outstanding state-action pairs from the entire state and action spaces by using random policies within a trust horizon, and then inherits the classical autoregressive-supervised mechanism during pretraining. To the best of our knowledge, this is the first work that enables effective ICRL under random policies and random contexts. We also establish quantitative analysis of the trustworthiness as well as the performance guarantees of SAD. Moreover, our empirical results across multiple popular ICRL benchmark environments demonstrate that, on average, SAD outperforms the best baseline by 236.3% in the offline evaluation and by 135.2% in the online evaluation.
Xi’an Jiaotong University（西安交通大学）；National University of Singapore（新加坡国立大学）
具有稀疏奖励的强化学习（RL）是一个主要挑战。我们提出了\emph{Hindsight Trust Region Policy Optimization}（HTRPO），一个新的RL算法，扩展了非常成功的TRPO算法与\emph{hindsight}，以解决稀疏奖励的挑战。后见之明指的是算法从跨目标的信息中学习的能力，包括那些不适用于当前任务的信息。HTRPO利用了两个主要思想。它引入了QKL，一个二次近似的KL散度约束的信赖域，导致减少KL散度估计的方差和改善的稳定性，在政策更新。它还提出了后见之明目标过滤（HGF）选择导电后见之明目标。在实验中，我们在各种稀疏奖励任务中评估HTRPO，包括简单的基准测试，基于图像的Atari游戏和模拟机器人控制。消融研究表明，QKL和HGF对学习稳定性和高绩效有很大贡献。比较结果表明，在所有的任务，HTRPO始终优于TRPO和HPG，一个国家的最先进的RL算法与稀疏奖励。
Local Guidance, Global Impact: Gaussian-Reshaped Trust Region Unlocks Behavior Transitions
Hong Kong University of Science and Technology（香港科技大学）；Mila - Québec AI Institute（魁北克AI研究院）；Université de Montréal（蒙特利尔大学）；Fudan University（复旦大学）；City University of Hong Kong（香港城市大学）
Dealing with Non-Stationarity in MARL via Trust-Region Decomposition
East China Normal University（华东师范大学）；The Chinese University of Hong Kong (Shenzhen)（香港中文大学（深圳））；Shenzhen Institute of Artificial Intelligence and Robotics for Society（深圳人工智能与机器人研究院）
Trust Region Inverse Reinforcement Learning: Explicit Dual Ascent using Local Policy Updates
Technical University of Darmstadt（达姆斯塔特技术大学）；Lund University（隆德大学）；German Research Center for AI (DFKI)（人工智能研究中心（DFKI））；Robotics Institute Germany (RIG)（德国机器人研究所（RIG））；Huawei, Noah’s Ark Lab（华为诺亚实验室）；University College London（伦敦大学学院）
Black-box Off-policy Estimation for Infinite-Horizon Reinforcement Learning
Google Research（谷歌研究院）；UT Austin（德克萨斯大学奥斯汀分校）
长期问题的非策略估计在许多现实应用中非常重要，例如医疗保健和机器人技术，其中高保真仿真器可能不可用，并且策略评估昂贵或不可能。最近，\cite{liu18breaking}提出了一种方法，避免了典型的基于重要性采样的方法所遭受的\emph{curse of horizon}。虽然显示出有希望的结果，但这种方法在实践中受到限制，因为它需要从\emph{known}行为策略的\emph{stationary distribution}中提取数据。在这项工作中，我们提出了一种新的方法，消除这种限制。特别是，我们制定的问题作为解决某一运营商的不动点。使用再生核希尔伯特空间（RKHS）的工具，我们开发了一种新的估计器，可以计算静态分布的重要性比，而不知道如何收集非策略数据。我们分析了它的渐近一致性和有限样本推广。在基准测试上的实验验证了该方法的有效性。
Embedding Safety into RL: A New Take on Trust Region Methods
Max Planck Institute for Human Cognitive and Brain Sciences（马克斯·普朗克人类认知与脑科学研究所）；Center for Scalable Data Analytics and Artificial Intelligence（可扩展数据分析与人工智能中心）
Value Enhancement of Reinforcement Learning via Efficient and Robust Trust Region Optimization
London School of Economics and Political Science（伦敦政治经济学院）；Shanghai University of Finance and Economics（上海财经大学）；The George Washington University（乔治华盛顿大学）
Trust-Calibrated VLA Policy Post-Training with Evolving Imagination
Sun Yat-sen University（中山大学）；Shenzhen Loop Area Institute（深圳河套学院）；The Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳））；Shanghai AI Laboratory（上海人工智能实验室）；University of Oxford（牛津大学）；University of Science and Technology of China（中国科学技术大学）
TrustMed-RL: Long-Horizon Reinforcement Learning for Evidence-Grounded Clinical Diagnosis
Baruch College（巴鲁克学院）；University of North Carolina at Chapel Hill（北卡罗来纳大学教堂山分校）；Tsinghua University（清华大学）；New York University（纽约大学）；Zhongshan Hospital (Xiamen), Fudan University（复旦大学附属中山医院（厦门））
