在强化学习中,选择何种数据或环境进行训练以提升下游性能是一个长期存在且备受关注的热点问题。尤其是无监督环境设计(Unsupervised Environment Design, UED)方法近年来受到广泛关注,其自适应课程有望让智能体对分布内和分布外任务都具备鲁棒性。本研究聚焦任务优先级度量指标,探究现有UED方法选择训练环境的机制。令人意外的是,尽管理论上这些方法以最大化遗憾(regret)为目标,但实际应用中的近似值与遗憾值并不相关,反而与成功率相关。这导致智能体的大量训练经验来自其已经掌握的环境,对能力提升几乎没有贡献。换言之,现有方法无法预测直观的“可学习性”度量指标,具体而言,它们无法持续识别出智能体有时能解决但并非总能解决的场景。基于上述分析,我们提出了一种直接在高可学习性场景上进行训练的方法。这种简单直观的方法在多个二元结果环境中表现优于现有UED方法,包括标准测试域Minigrid,以及一个受真实机器人问题启发而设计的新场景。我们还提出了一种新的对抗性评估流程,用于直接度量鲁棒性,该流程与条件风险价值(conditional value at risk, CVaR)高度契合。我们已将所有代码开源,并在以下地址提供最终策略的可视化结果:https://github.com/amacrutherford/sampling-for-learnability。
英文摘要:
What data or environments to use for training to improve downstream performance is a longstanding and very topical question in reinforcement learning. In particular, Unsupervised Environment Design (UED) methods have gained recent attention as their adaptive curricula promise to enable agents to be robust to in- and out-of-distribution tasks. This work investigates how existing UED methods select training environments, focusing on task prioritisation metrics. Surprisingly, despite methods aiming to maximise regret in theory, the practical approximations do not correlate with regret but with success rate. As a result, a significant portion of an agent's experience comes from environments it has already mastered, offering little to no contribution toward enhancing its abilities. Put differently, current methods fail to predict intuitive measures of ``learnability.'' Specifically, they are unable to consistently identify those scenarios that the agent can sometimes solve, but not always. Based on our analysis, we develop a method that directly trains on scenarios with high learnability. This simple and intuitive approach outperforms existing UED methods in several binary-outcome environments, including the standard domain of Minigrid and a novel setting closely inspired by a real-world robotics problem. We further introduce a new adversarial evaluation procedure for directly measuring robustness, closely mirroring the conditional value at risk (CVaR). We open-source all our code and present visualisations of final policies here: https://github.com/amacrutherford/sampling-for-learnability.
Investigating Value of Curriculum Reinforcement Learning in Autonomous Driving Under Diverse Road and Weather Conditions
Istanbul Technical University(伊斯坦布尔理工大学);ITU Artificial Intelligence and Data Science Research Center(伊斯坦布尔理工大学人工智能与数据科学研究中心);Bogazici University(博阿齐奇大学)
Adversarial Environment Design via Regret-Guided Diffusion Models
Seoul National University(首尔大学);ASRI(先进科学技术研究院);Interdisciplinary Program in Artificial Intelligence(人工智能跨学科项目);Department of Electrical and Computer Engineering(电气与计算机工程系)
FOSSIL: Regret-Minimizing Curriculum Learning for Metadata-Free and Low-Data Mpox Diagnosis
Computer Science Division, Gwinnett Technical College, Lawrenceville, GA, USA(基普顿技术学院计算机科学系);Department of Computer Science, University of Kentucky, Lexington, KY, USA(肯塔基大学计算机科学系);Department of Biomedical Engineering, Kumoh National Institute of Technology, Gumi, South Korea(Kumoh国立技术学院生物医学工程系);Department of Biology, Georgia Institute of Technology, Atlanta, GA, USA(佐治亚理工学院生物学系);Oaro Dermatology Center, Seongnam, Gyeonggi-do, South Korea(Oaro皮肤科中心)
Actor-Curator: Co-adaptive Curriculum Learning via Policy-Improvement Bandits for RL Post-Training
University of Illinois Chicago(伊利诺伊大学香槟分校);Caltech(加州理工学院);RPI(罗切斯特理工学院);MBZUAI(澳门大学人工智能研究院);University of Chicago(芝加哥大学);NEC Laboratories America(NEC美国实验室)
Estimating the Effects of Sample Training Orders for Large Language Models without Retraining
Gaoling School of Artificial Intelligence, Renmin University of China(中国人民大学耿丽人工智能学院);Center for Data Science, Peking University(北京大学数据科学中心);School of Engineering Mathematics and Technology, University of Bristol(布里斯托大学工程数学与技术学院);Department of Machine Learning, Mohamed bin Zayed University of Artificial Intelligence(穆罕默德·本·扎耶德人工智能大学机器学习系)
On Improving Model-Free Algorithms for Decentralized Multi-Agent Reinforcement Learning
University of Illinois Urbana-Champaign(伊利诺伊大学厄巴纳-香槟分校);University of California, Los Angeles(加利福尼亚大学洛杉矶分校);DeepMind(深度思维);Massachusetts Institute of Technology(麻省理工学院)
多智能体强化学习(MARL)算法经常遭受指数样本复杂度对智能体数量的依赖,这种现象被称为\emph{the curse of multiagents}。在本文中,我们解决这一挑战,调查样本效率的无模型算法在\emph{decentralized} MARL,并旨在改善现有的算法沿这条线。对于学习(粗)一般和马尔可夫博弈中的相关均衡,我们提出了\emph{stage-based} V-学习算法,大大简化了最近的作品的算法设计和分析,并规避了一个相当复杂的无\emph{weighted}遗憾强盗子程序。针对马尔可夫势博弈中纳什均衡的学习问题,提出了一种独立的策略梯度算法,该算法采用分散的基于动量的方差缩减技术。我们所有的算法都是去中心化的,因为每个代理只能根据其本地信息做出决策。在学习过程中既不需要通信,也不需要集中协调,从而导致对大量代理的自然泛化。我们还提供了数值模拟来证实我们的理论研究结果。