DRED：通过数据正则化环境设计实现强化学习中的零样本迁移
DRED: Zero-Shot Transfer in Reinforcement Learning via Data-Regularised Environment Design
AI总结：
本文针对强化学习零样本泛化问题，分析自适应关卡采样的正则化机理及 UED 的分布偏移缺陷，并提出通过生成模型锚定真实关卡参数分布的 DRED，显著提升零样本迁移性能。
摘要：
使用深度强化学习（RL）训练的自主智能体往往缺乏成功泛化到新环境的能力，即使这些环境与其在训练期间遇到的环境具有共同特征。在本工作中，我们研究了单个环境实例（即关卡）的采样如何影响 RL 智能体的零样本泛化（ZSG）能力。我们发现，对于共享基础层的深度 actor-critic 架构，根据关卡的价值损失对其进行优先排序，会使所生成训练数据中智能体内部表示与训练关卡集合之间的互信息最小化。这为某些自适应采样策略所实现的正则化提供了一种新的理论依据。随后，我们将注意力转向无监督环境设计（UED）方法，这类方法假定可以控制关卡生成。我们发现，现有 UED 方法可能显著改变训练分布，从而导致较低的 ZSG 性能。为同时防止过拟合和分布偏移，我们提出了数据正则化环境设计（DRED）。DRED 使用一个生成模型来生成关卡，该模型经过训练以近似初始关卡参数集合的真实分布。通过这种基于真实数据的锚定，DRED 在 ZSG 方面相较自适应关卡采样策略和 UED 方法取得了显著提升。我们的代码和实验数据可在 https://github.com/uoe-agents/dred 获取。
英文摘要：
Autonomous agents trained using deep reinforcement learning (RL) often lack the ability to successfully generalise to new environments, even when these environments share characteristics with the ones they have encountered during training. In this work, we investigate how the sampling of individual environment instances, or levels, affects the zero-shot generalisation (ZSG) ability of RL agents. We discover that, for deep actor-critic architectures sharing their base layers, prioritising levels according to their value loss minimises the mutual information between the agent's internal representation and the set of training levels in the generated training data. This provides a novel theoretical justification for the regularisation achieved by certain adaptive sampling strategies. We then turn our attention to unsupervised environment design (UED) methods, which assume control over level generation. We find that existing UED methods can significantly shift the training distribution, which translates to low ZSG performance. To prevent both overfitting and distributional shift, we introduce data-regularised environment design (DRED). DRED generates levels using a generative model trained to approximate the ground truth distribution of an initial set of level parameters. Through its grounding, DRED achieves significant improvements in ZSG over adaptive level sampling strategies and UED methods. Our code and experimental data are available at https://github.com/uoe-agents/dred.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
