使用强化学习(RL)训练具备通用能力的智能体仍是一项重大挑战。提升RL智能体鲁棒性的一条有前景途径是使用课程。其中一类方法将环境设计构建为学生与教师之间的博弈,采用基于遗憾的目标,在学生智能体能力边界上生成环境实例(即关卡)。这些方法具有通用性优势,并在均衡状态下具备理论保证,但在具有挑战性的设计空间中往往难以找到有效关卡。相比之下,演化方法试图渐进式改变环境复杂度,从而实现潜在的开放式学习,但通常依赖特定领域的启发式规则和大量计算资源。本文提出在有原则的、基于遗憾的课程中利用演化的力量。我们的方法称为通过编辑关卡进行对抗性复杂度累积(Adversarially Compounding Complexity by Editing Levels,ACCEL),旨在持续在智能体能力边界上生成关卡,使课程从简单开始并逐渐变得复杂。ACCEL保留了以往基于遗憾方法的理论优势,同时在多种不同环境中带来显著的实验性能提升。本文的交互式版本可在accelagent.github.io获取。
英文摘要:
It remains a significant challenge to train generally capable agents with reinforcement learning (RL). A promising avenue for improving the robustness of RL agents is through the use of curricula. One such class of methods frames environment design as a game between a student and a teacher, using regret-based objectives to produce environment instantiations (or levels) at the frontier of the student agent's capabilities. These methods benefit from their generality, with theoretical guarantees at equilibrium, yet they often struggle to find effective levels in challenging design spaces. By contrast, evolutionary approaches seek to incrementally alter environment complexity, resulting in potentially open-ended learning, but often rely on domain-specific heuristics and vast amounts of computational resources. In this paper we propose to harness the power of evolution in a principled, regret-based curriculum. Our approach, which we call Adversarially Compounding Complexity by Editing Levels (ACCEL), seeks to constantly produce levels at the frontier of an agent's capabilities, resulting in curricula that start simple but become increasingly complex. ACCEL maintains the theoretical benefits of prior regret-based methods, while providing significant empirical gains in a diverse set of environments. An interactive version of the paper is available at accelagent.github.io.
On Mitigating Affinity Bias through Bandits with Evolving Biased Feedback
Algorithms and Randomness Center, Georgia Institute of Technology(佐治亚理工学院算法与随机性中心);The University of Texas at Austin(德克萨斯大学奥斯汀分校);Google Research(谷歌研究院)
The Non-Bayesian Restless Multi-Armed Bandit: A Case of Near-Logarithmic Strict Regret
Department of Aeronautics;Astronautics Massachusetts Institute of Technology, Cambridge, MA 02139, USA Ming Hsieh Department of Electrical Engineering University of Southern California, Los Angeles, CA 90089, USA Department of Electrical;Computer Engineering University of California, Davis, CA 95616, USA Email
Department of Applied Mathematics and Theoretical Physics, University of Cambridge(应用数学与理论物理系,剑桥大学);Centre for Human-Inspired Artificial Intelligence, University of Cambridge(启发式人工智能中心,剑桥大学);African Institute for Mathematical Sciences, South Africa(南非数学科学研究所);Department of Engineering Science, University of Oxford(工程科学系,牛津大学)
Active Zero: Self-Evolving Vision-Language Models through Active Environment Exploration
Foundation Model Research Center, Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所基础模型研究中心);School of Artificial Intelligence, University of Chinese Academy of Sciences(中国科学院大学人工智能学院);National University of Singapore(新加坡国立大学);Wuhan AI Research(武汉人工智能研究所);Tsinghua University(清华大学)
CASCADE: Case-Based Continual Adaptation for Large Language Models During Deployment
School of Artificial Intelligence, Jilin University(吉林大学人工智能学院);Engineering Research Center of Knowledge-Driven Human-Machine Intelligence, Jilin University(吉林大学知识驱动人机智能工程研究中心);International Center of Future Science, Jilin University(吉林大学未来科学国际中心);Department of Informatics, King’s College London(伦敦国王学院信息学院);The Alan Turing Institute(艾伦·图灵研究所);AI Centre, Department of Computer Science, UCL(UCL计算机科学系人工智能中心)
Adapting to Fragmented and Evolving Data: A Fisher Information Perspective
School of Mathematics and Computer Science, Institute of Business Administration(数学与计算机科学学院,商业行政研究所);National University of Computer and Emerging Sciences(计算机与新兴科学国立大学)
False Frontiers: Diagnosing and Mitigating Co-Cheating in Self-Evolving Search Agents
Rutgers University(罗格斯大学);University of California, San Diego(加州大学圣迭戈分校);University of Michigan(密歇根大学);McGill University(麦吉尔大学);King Fahd University of Petroleum and Minerals(法赫德国王石油矿产大学)