Generalization and sample efficiency have been long-standing issues concerning reinforcement learning, and thus the field of Offline Meta-Reinforcement Learning~(OMRL) has gained increasing attention due to its potential of solving a wide range of problems with static and limited offline data. Existing OMRL methods often assume sufficient training tasks and data coverage to apply contrastive learning to extract task representations. However, such assumptions are not applicable in several real-world applications and thus undermine the generalization ability of the representations. In this paper, we consider OMRL with two types of data limitations: limited training tasks and limited behavior diversity and propose a novel algorithm called GENTLE for learning generalizable task representations in the face of data limitations. GENTLE employs Task Auto-Encoder~(TAE), which is an encoder-decoder architecture to extract the characteristics of the tasks. Unlike existing methods, TAE is optimized solely by reconstruction of the state transition and reward, which captures the generative structure of the task models and produces generalizable representations when training tasks are limited. To alleviate the effect of limited behavior diversity, we consistently construct pseudo-transitions to align the data distribution used to train TAE with the data distribution encountered during testing. Empirically, GENTLE significantly outperforms existing OMRL methods on both in-distribution tasks and out-of-distribution tasks across both the given-context protocol and the one-shot protocol.
Provably Improved Context-Based Offline Meta-RL with Attention and Contrastive Learning
Tencent AI Lab(腾讯AI实验室);Megvii Research Shanghai(上海旷视研究院);Tsinghua University(清华大学);Peking University(北京大学);University of Texas at Arlington(德克萨斯大学阿灵顿分校)
STA-GANN: A Valid and Generalizable Spatio-Temporal Kriging Approach
State Key Laboratory of AI Safety, Institute of Computing Technology, Chinese Academy of Sciences(人工智能安全国家重点实验室,计算技术研究所,中国科学院);School of Aerospace Engineering, Beijing Institute of Technology(航空工程学院,北京理工大学)
A generalizable framework for unlocking missing reactions in genome-scale metabolic networks using deep learning
University of South Carolina(南卡罗来纳大学);Central South University(中南大学);Peking University(北京大学);Yangtze Delta Region Institute, University of Electronic Science and Technology of China(电子科技大学长三角研究院);National Technology Innovation Center of Synthetic Biology(国家合成生物技术创新中心);Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences(中国科学院深圳先进技术研究院)
Disentangling Policy from Offline Task Representation Learning via Adversarial Data Augmentation
National Key Laboratory for Novel Software Technology(计算机软件新技术国家重点实验室);School of Artificial Intelligence(人工智能学院);Nanjing University(南京大学);Polixir Technologies(铂乐奇科技)
VLGOR: Visual-Language Knowledge Guided Offline Reinforcement Learning for Generalizable Agents
National Key Laboratory for Novel Software Technology, Nanjing University, China \& School of Artificial Intelligence, Nanjing University, China;Computational Intelligence Center (CIC), School of Computer;Artificial Intelligence, Shandong Jianzhu University, Jinan, 250101, China
Contextual Transformer for Offline Meta Reinforcement Learning
University of Chinese Academy of Sciences(中国科学院大学);Peking University(北京大学);University College London(伦敦大学学院);Chinese Academy of Sciences(中国科学院);King’s College London(伦敦国王学院)
Generalizable Heuristic Generation Through LLMs with Meta-Optimization
Nanyang Technological University(南洋理工大学);Shandong University(山东大学);Singapore Management University(新加坡国立大学);Eindhoven University of Technology(埃因霍温理工大学)
Context Shift Reduction for Offline Meta-Reinforcement Learning
University of Science and Technology of China(中国科学技术大学);Cambricon Technologies(寒武纪科技);University of Chinese Academy of Sciences(中国科学院大学);Intelligent Software Research Center, Institute of Software, CAS(中国科学院软件研究所智能软件研究中心);Shanghai Innovation Center(上海创新中心)
MetaToolAgent: Towards Generalizable Tool Usage in LLMs through Meta-Learning
College of Informatics, Huazhong Agricultural University(华中农业大学信息学院);Industrial AI Research Centre, University of South Australia(南澳大利亚大学工业AI研究中心);School of Computer Science, Wuhan University(武汉大学计算机学院)
MetaDrive: Composing Diverse Driving Scenarios for Generalizable Reinforcement Learning
ETH Zurich(苏黎世联邦理工学院);The Chinese University of Hong Kong(香港中文大学);University of California, Los Angeles(加利福尼亚大学洛杉矶分校);Centre for Perceptual and Interactive Intelligence(感知与交互智能研究中心)
Generalizable Black-Box Adversarial Attack with Meta Learning
Tsinghua University(清华大学);Chinese University of Hong Kong, Shenzhen(香港中文大学(深圳));Tencent AI Lab(腾讯人工智能实验室);University of Electronic Science and Technology of China(电子科技大学)
Enhancing Online Reinforcement Learning with Meta-Learned Objective from Offline Data
University of Electronic Science and Technology of China(电子科技大学);Sichuan Artificial Intelligence Research Institute(四川省人工智能研究院);Duke Kunshan University(昆山杜克大学)
Your Offline Policy is Not Trustworthy: Bilevel Reinforcement Learning for Sequential Portfolio Optimization
MoE Key Lab of Artificial Intelligence, AI Institute, Shanghai Jiao Tong University, China(人工智能教育部重点实验室、人工智能研究院、上海交通大学);Department of Computer Science, University of Illinois Chicago, USA(计算机科学系、伊利诺伊大学芝加哥分校)