arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2408.15099cs.LGcs.AIcs.RO
收藏

无遗憾:研究并改进课程发现中的遗憾近似方法

No Regrets: Investigating and Improving Regret Approximations for Curriculum Discovery

University of Oxford(牛津大学)

Alexander Rutherford, Michael Beukman, Timon Willi, Bruno Lacerda, Nick Hawes, Jakob Foerster

AI总结:

本研究发现现有无监督环境设计(UED)方法的遗憾近似值与实际遗憾不相关,提出直接在高可学习性场景训练的新方法,在多个环境中性能更优,还引入了基于CVaR的对抗性鲁棒性评估流程并开源代码。

摘要:

在强化学习中,选择何种数据或环境进行训练以提升下游性能是一个长期存在且备受关注的热点问题。尤其是无监督环境设计(Unsupervised Environment Design, UED)方法近年来受到广泛关注,其自适应课程有望让智能体对分布内和分布外任务都具备鲁棒性。本研究聚焦任务优先级度量指标,探究现有UED方法选择训练环境的机制。令人意外的是,尽管理论上这些方法以最大化遗憾(regret)为目标,但实际应用中的近似值与遗憾值并不相关,反而与成功率相关。这导致智能体的大量训练经验来自其已经掌握的环境,对能力提升几乎没有贡献。换言之,现有方法无法预测直观的“可学习性”度量指标,具体而言,它们无法持续识别出智能体有时能解决但并非总能解决的场景。基于上述分析,我们提出了一种直接在高可学习性场景上进行训练的方法。这种简单直观的方法在多个二元结果环境中表现优于现有UED方法,包括标准测试域Minigrid,以及一个受真实机器人问题启发而设计的新场景。我们还提出了一种新的对抗性评估流程,用于直接度量鲁棒性,该流程与条件风险价值(conditional value at risk, CVaR)高度契合。我们已将所有代码开源,并在以下地址提供最终策略的可视化结果:https://github.com/amacrutherford/sampling-for-learnability。

英文摘要:

What data or environments to use for training to improve downstream performance is a longstanding and very topical question in reinforcement learning. In particular, Unsupervised Environment Design (UED) methods have gained recent attention as their adaptive curricula promise to enable agents to be robust to in- and out-of-distribution tasks. This work investigates how existing UED methods select training environments, focusing on task prioritisation metrics. Surprisingly, despite methods aiming to maximise regret in theory, the practical approximations do not correlate with regret but with success rate. As a result, a significant portion of an agent's experience comes from environments it has already mastered, offering little to no contribution toward enhancing its abilities. Put differently, current methods fail to predict intuitive measures of ``learnability.'' Specifically, they are unable to consistently identify those scenarios that the agent can sometimes solve, but not always. Based on our analysis, we develop a method that directly trains on scenarios with high learnability. This simple and intuitive approach outperforms existing UED methods in several binary-outcome environments, including the standard domain of Minigrid and a novel setting closely inspired by a real-world robotics problem. We further introduce a new adversarial evaluation procedure for directly measuring robustness, closely mirroring the conditional value at risk (CVaR). We open-source all our code and present visualisations of final policies here: https://github.com/amacrutherford/sampling-for-learnability.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑