Large language models (LLMs) are increasingly trained in complex Reinforcement Learning, multi-agent environments, making it difficult to understand how behavior changes over training. Sparse Autoencoders (SAEs) have recently shown to be useful for data-centric interpretability. In this work, we analyze large-scale reinforcement learning training runs from the sophisticated environment of Full-Press Diplomacy by applying pretrained SAEs, alongside LLM-summarizer methods. We introduce Meta-Autointerp, a method for grouping SAE features into interpretable hypotheses about training dynamics. We discover fine-grained behaviors including role-playing patterns, degenerate outputs, language switching, alongside high-level strategic behaviors and environment-specific bugs. Through automated evaluation, we validate that 90% of discovered SAE Meta-Features are significant, and find a surprising reward hacking behavior. However, through two user studies, we find that even subjectively interesting and seemingly helpful SAE features may be worse than useless to humans, along with most LLM generated hypotheses. However, a subset of SAE-derived hypotheses are predictively useful for downstream tasks. We further provide validation by augmenting an untrained agent's system prompt, improving the score by +14.2%. Overall, we show that SAEs and LLM-summarizer provide complementary views into agent behavior, and together our framework forms a practical starting point for future data-centric interpretability work on ensuring trustworthy LLM behavior throughout training.
Agent-Centric Representations for Multi-Agent Reinforcement Learning
DeepMind；Google Research（谷歌研究院）
以对象为中心的表示最近在处理关系推理任务方面取得了重大进展。通过在神经架构中构建强的以对象为中心的归纳偏差，最近的努力已经提高了这些问题的机器学习算法的泛化和数据效率。涉及关系推理的一个问题类别仍然未得到充分探讨，那就是多智能体强化学习（MARL）。在这里，我们调查是否以对象为中心的表示也是有益的，在充分合作的MARL设置。具体来说，我们研究了两种方法，将代理为中心的归纳偏见到我们的RL算法：1。引入一个以代理为中心的注意力模块，并在代理之间建立显式连接2。添加以代理为中心的无监督预测目标（即不使用动作标签），用作MARL的辅助损失，或作为预训练步骤的基础。我们在Google Research Football环境和DeepMind Lab 2D上评估了这些方法。经验上，以主体为中心的表示学习导致主体之间更复杂的合作策略的出现，以及增强的样本效率和泛化能力。
Deep Reinforcement Learning via Object-Centric Attention
Technical University of Darmstadt（达姆施塔特工业大学）；Hessian Center for Artificial Intelligence (hessian.AI)（黑森人工智能中心）；Sorbonne Université（索邦大学）；German Research Center for Artificial Intelligence (DFKI)（德国人工智能研究中心）；Centre for Cognitive Science, Darmstadt, Germany（达姆施塔特认知科学中心）
Technical University of Darmstadt（达姆施塔特工业大学）；Hessian Center for Artificial Intelligence (hessian.AI)（黑森人工智能中心）；Centre for Cognitive Science of Darmstadt（达姆施塔特认知科学中心）；German Research Center for Artificial Intelligence (DFKI)（德国人工智能研究中心）
There is no Accuracy-Interpretability Tradeoff in Reinforcement Learning for Mazes
Tel Aviv University（特拉维夫大学）；Google Research（谷歌研究院）；Duke University（杜克大学）
可解释性是强化学习系统中可信度的重要组成部分。然而，可解释性可能以性能恶化为代价，导致许多研究人员构建复杂的模型。我们的目标是分析可解释性的成本。我们表明，在某些情况下，可以实现政策的可解释性，同时保持其最优性。我们专注于一个经典的问题，从强化学习：迷宫与$k$障碍$\mathbb{R}^d$。我们证明了一个小的决策树的存在性与线性函数在每个内部节点和深度$O(\log k + 2^d)$，代表一个最优的政策。注意，对于$d$恒定的有趣情况，我们有$O(\log k)$深度。因此，在这种情况下，不存在准确性-可解释性权衡。为了证明这一结果，我们使用了一种新的“压缩”技术，它可能在其他设置中有用。
Interpretability by Design for Efficient Multi-Objective Reinforcement Learning
School of Informatics, The University of Edinburgh（信息学院，爱丁堡大学）；School of Mathematics, The University of Edinburgh（数学学院，爱丁堡大学）；Edinburgh Centre for Robotics（爱丁堡机器人中心）
Towards Control-Centric Representations in Reinforcement Learning from Images
Beijing Institute of Electronic System Engineering（北京电子系统工程研究所）；Peking University（北京大学）；Beijing Institute of Technology（北京理工大学）；University of the Sunshine Coast（阳光海岸大学）
本文提出 ReBis，通过 Transformer 隐式建模动态并结合互模拟损失与非对称重构损失，在稀疏奖励下学习以控制为中心的图像强化学习表示，并在 Atari 和 DeepMind Control Suit 上验证了其优越性。
A Survey of Reinforcement Learning for Large Language Models under Data Scarcity: Challenges and Solutions
Peking University（北京大学）；Shanghai Artificial Intelligence Laboratory（上海人工智能实验室）；National University of Singapore（新加坡国立大学）；Nankai University（南开大学）；Tongji University（同济大学）；Nanjing University（南京大学）；University of Wollongong（沃林根大学）；Shanghai Jiao Tong University（上海交通大学）；University of Wisconsin-Madison（威斯康星大学麦迪逊分校）
Learning Interactive World Model for Object-Centric Reinforcement Learning
University of California San Diego（加州大学圣地亚哥分校）；Mohamed bin Zayed University of Artificial Intelligence（马尔代夫穆罕默德·本·扎耶德人工智能大学）；University of Amsterdam（阿姆斯特丹大学）
Human-Centric Aware UAV Trajectory Planning in Search and Rescue Missions Employing Multi-Objective Reinforcement Learning with AHP and Similarity-Based Experience Replay
Interdisciplinary Centre for Security, Reliability and Trust (SnT), University of Luxembourg（卢森堡大学安全、可靠性与信任跨学科中心）；University of Luxembourg（卢森堡大学）
Local Policy Optimization for Trajectory-Centric Reinforcement Learning
UTA Research Institute, University of Texas at Arlington（德克萨斯大学阿灵顿分校UTA研究所）；Mitsubishi Electric Research Laboratories (MERL)（三菱电机研究实验室（MERL））；University of Texas at Arlington（德克萨斯大学阿灵顿分校）
Balancing Interpretability and Performance in Reinforcement Learning: An Adaptive Spectral Based Linear Approach
Center for Intelligent Decision Making and Machine Learning, School of Management, Xi’an Jiaotong University, Xi’an, China（智能决策与机器学习中心，管理学院，西安交通大学）；Department of Mathematics, Hong Kong Baptist University, Kowloon, Hong Kong（数学系，香港 Baptist 大学）
Human-Centric Traffic Signal Control for Equity: A Multi-Agent Action Branching Deep Reinforcement Learning Approach
Department of Infrastructure Engineering, Faculty of Engineering and Information Technology, The University of Melbourne, VIC 3010, Australia（工程与信息技术学院基础设施工程系，墨尔本大学）
OpenRL: A Unified Reinforcement Learning Framework
Paradigm Inc.（第四范式）；Robotics Institute, Carnegie Mellon University（卡内基梅隆大学机器人研究所）；School of Data Science, Fudan University（复旦大学大数据学院）；Beijing University of Posts and Telecommunications（北京邮电大学）
