Reinforced Imitation in Heterogeneous Action Space
Jagiellonian University（雅盖隆大学）；Element AI；Université de Montréal（蒙特利尔大学）；MILA（蒙特利尔学习算法研究所）；Rutgers University（罗格斯大学）
Konrad Zolna, Negar Rostamzadeh, Yoshua Bengio, Sungjin Ahn, Pedro O. Pinheiro
备注：
The extended version of the work "Reinforced Imitation Learning from Observations" presented on the NeurIPS workshop "Imitation Learning and its Challenges in Robotics"
Imitation learning is an effective alternative approach to learn a policy when the reward function is sparse. In this paper, we consider a challenging setting where an agent and an expert use different actions from each other. We assume that the agent has access to a sparse reward function and state-only expert observations. We propose a method which gradually balances between the imitation learning cost and the reinforcement learning objective. In addition, this method adapts the agent's policy based on either mimicking expert behavior or maximizing sparse reward. We show, through navigation scenarios, that (i) an agent is able to efficiently leverage sparse rewards to outperform standard state-only imitation learning, (ii) it can learn a policy even when its actions are different from the expert, and (iii) the performance of the agent is not bounded by that of the expert, due to the optimized usage of sparse rewards.
Asian Institute of Technology（亚洲理工学院）；The University of Tokyo（东京大学）
Compared to traditional imitation learning methods such as DAgger and DART, intervention-based imitation offers a more convenient and sample efficient data collection process to users. In this paper, we introduce Reinforced Intervention-based Learning (ReIL), a framework consisting of a general intervention-based learning algorithm and a multi-task imitation learning model aimed at enabling non-expert users to train agents in real environments with little supervision or fine tuning. ReIL achieves this with an algorithm that combines the advantages of imitation learning and reinforcement learni…
Reinforcement Learning (RL) requires a large amount of exploration especially in sparse-reward settings. Imitation Learning (IL) can learn from expert demonstrations without exploration, but it never exceeds the expert's performance and is also vulnerable to distributional shift between demonstration and execution. In this paper, we radically unify RL and IL based on Free Energy Principle (FEP). FEP is a unified Bayesian theory of the brain that explains perception, action and model learning by a common fundamental principle. We present a theoretical extension of FEP and derive an algorithm in…
University of Texas at Austin（德克萨斯大学奥斯汀分校）；University of Edinburgh（爱丁堡大学）；University of Wisconsin-Madison（威斯康星大学麦迪逊分校）；Army Research Laboratory（美国陆军研究实验室）；Sony AI（索尼人工智能）
Augmenting reinforcement learning with imitation learning is often hailed as a method by which to improve upon learning from scratch. However, most existing methods for integrating these two techniques are subject to several strong assumptions---chief among them that information about demonstrator actions is available. In this paper, we investigate the extent to which this assumption is necessary by introducing and evaluating reinforced inverse dynamics modeling (RIDM), a novel paradigm for combining imitation from observation (IfO) and reinforcement learning with no dependence on demonstrator…
University of Central Florida（中佛罗里达大学）；University of Macau（澳门大学）；Rutgers University（罗格斯大学）
Mobile user profiling refers to the efforts of extracting users' characteristics from mobile activities. In order to capture the dynamic varying of user characteristics for generating effective user profiling, we propose an imitation-based mobile user profiling framework. Considering the objective of teaching an autonomous agent to imitate user mobility based on the user's profile, the user profile is the most accurate when the agent can perfectly mimic the user behavior patterns. The profiling framework is formulated into a reinforcement learning task, where an agent is a next-visit planner, …
Going Beyond Expert Performance via Deep Implicit Imitation Reinforcement Learning
Department of Physics, J.K. Institute of Science（J.K.科学研究院物理系）；World Scientific University（世界科学大学）；University of Intelligent Studies（智能研究大学）；Technical University of Crete（克里特技术大学）
We present a novel Neural Embedding Spatio-Temporal (NEST) point process model for spatio-temporal discrete event data and develop an efficient imitation learning (a type of reinforcement learning) based approach for model fitting. Despite the rapid development of one-dimensional temporal point processes for discrete event data, the study of spatial-temporal aspects of such data is relatively scarce. Our model captures complex spatio-temporal dependence between discrete events by carefully design a mixture of heterogeneous Gaussian diffusion kernels, whose parameters are parameterized by neura…
ANEO；Centre Borelli - ENS Paris-saclay（巴黎萨克雷高等师范学院博雷利中心）；Criteo AI Lab（科韬人工智能实验室）；Crest, ENSAE（法国国立统计与经济管理学院 Crest 研究中心）
The objective of offline RL is to learn optimal policies when a fixed exploratory demonstrations data-set is available and sampling additional observations is impossible (typically if this operation is either costly or rises ethical questions). In order to solve this problem, off the shelf approaches require a properly defined cost function (or its evaluation on the provided data-set), which are seldom available in practice. To circumvent this issue, a reasonable alternative is to query an expert for few optimal demonstrations in addition to the exploratory data-set. The objective is then to l…
Behavior cloning (BC) is often practical for robot learning because it allows a policy to be trained offline without rewards, by supervised learning on expert demonstrations. However, BC does not effectively leverage what we will refer to as unlabeled experience: data of mixed and unknown quality without reward annotations. This unlabeled data can be generated by a variety of sources such as human teleoperation, scripted policies and other agents on the same robot. Towards data-driven offline robot learning that can use this unlabeled experience, we introduce Offline Reinforced Imitation Learn…
ReinDSplit: Reinforced Dynamic Split Learning for Pest Recognition in Precision Agriculture
Department of Computer Science, Missouri University of Science and Technology, USA（计算机科学系，密苏里科技大学）；Department of Mechanical Engineering, Iowa State University, USA（机械工程系，爱荷华州立大学）；Department of Agronomy, Iowa State University, USA（农业学系，爱荷华州立大学）
Tsinghua University（清华大学）；Amazon（亚马逊）；University of Electronic Science and Technology of China（电子科技大学）；The Chinese University of Hong Kong（香港中文大学）；University of Illinois at Chicago（伊利诺伊大学芝加哥分校）
Information Extraction (IE) aims to extract structured information from heterogeneous sources. IE from natural language texts include sub-tasks such as Named Entity Recognition (NER), Relation Extraction (RE), and Event Extraction (EE). Most IE systems require comprehensive understandings of sentence structure, implied semantics, and domain knowledge to perform well; thus, IE tasks always need adequate external resources and annotations. However, it takes time and effort to obtain more human annotations. Low-Resource Information Extraction (LRIE) strives to use unsupervised data, reducing the …
The University of Tokyo（东京大学）；RIKEN Center for Advanced Intelligence Project（理化学研究所先进智能项目中心）；ZheJiang University（浙江大学）；Nanjing University（南京大学）
In many real-world imitation learning tasks, the demonstrator and the learner have to act under different observation spaces. This situation brings significant obstacles to existing imitation learning approaches, since most of them learn policies under homogeneous observation spaces. On the other hand, previous studies under different observation spaces have strong assumptions that these two observation spaces coexist during the entire learning process. However, in reality, the observation coexistence will be limited due to the high cost of acquiring expert observations. In this work, we study…
Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences（中国科学院深圳先进技术研究院）；University of Chinese Academy of Sciences（中国科学院大学）；Hong Kong University of Science and Technology（香港科技大学）
Humans are capable of learning a new behavior by observing others perform the skill. Robots can also implement this by imitation learning. Furthermore, if with external guidance, humans will master the new behavior more efficiently. So how can robots implement this? To address the issue, we present Federated Imitation Learning (FIL) in the paper. Firstly, a knowledge fusion algorithm deployed on the cloud for fusing knowledge from local robots is presented. Then, effective transfer learning methods in FIL are introduced. With FIL, a robot is capable of utilizing knowledge from other robots to …
Hybrid Orchestration of Edge AI and Microservices via Graph-based Self-Imitation Learning
School of Computer Science, South-Central Minzu University（中央民族大学南中央学院）；Key Laboratory of Cyber-Physical Fusion Intelligent Computing, State Ethnic Affairs Commission（国家民族事务委员会网络物理融合智能计算重点实验室）；Hubei Key Laboratory of Smart Internet Technology, School of Electronic Information and Communication, Huazhong University of Science and Technology（湖北智能互联网技术重点实验室，华中科技大学电子信息与通信学院）；Centre for Frontier AI Research, Agency for Science, Technology and Research (A*STAR)（前沿人工智能研究中心，科技研究局（A*STAR））
Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences（中国科学院深圳先进技术研究院）；University of Chinese Academy of Sciences（中国科学院大学）；Hong Kong University of Science and Technology（香港科技大学）；University of Macau（澳门大学）
Humans are capable of learning a new behavior by observing others to perform the skill. Similarly, robots can also implement this by imitation learning. Furthermore, if with external guidance, humans can master the new behavior more efficiently. So, how can robots achieve this? To address the issue, we present a novel framework named FIL. It provides a heterogeneous knowledge fusion mechanism for cloud robotic systems. Then, a knowledge fusion algorithm in FIL is proposed. It enables the cloud to fuse heterogeneous knowledge from local robots and generate guide models for robots with service r…
MAVEN-T: Reinforced Heterogeneous Distillation for Real-Time Multi-Agent Trajectory Prediction
School of Mathematical Sciences, Shanghai Jiao Tong University（上海交通大学数学科学学院）；Bio-X Institutes, Key Laboratory for the Genetics of Developmental and Neuropsychiatric Disorders, Shanghai Jiao Tong University（上海交通大学Bio-X研究院、发育与神经精神疾病遗传学重点实验室）；Shanghai Key Laboratory of Psychotic Disorders, Brain Science and Technology Research Center, Shanghai Jiao Tong University（上海精神疾病重点实验室、脑科学与技术研究中心，上海交通大学）
Providing a suitable reward function to reinforcement learning can be difficult in many real world applications. While inverse reinforcement learning (IRL) holds promise for automatically learning reward functions from demonstrations, several major challenges remain. First, existing IRL methods learn reward functions from scratch, requiring large numbers of demonstrations to correctly infer the reward for each task the agent may need to perform. Second, existing methods typically assume homogeneous demonstrations for a single behavior or task, while in practice, it might be easier to collect d…
MARS$^2$: Scaling Multi-Agent Tree Search via Reinforcement Learning for Code Generation
Shanghai Artificial Intelligence Laboratory（上海人工智能实验室）；Shanghai Innovation Institute（上海创新研究院）；Harbin Institute of Technology（哈尔滨工业大学）；Tsinghua University（清华大学）
MARTI-MARS$^2$: Scaling Multi-Agent Self-Search via Reinforcement Learning for Code Generation
Shanghai AI Laboratory（上海人工智能实验室）；Tsinghua University（清华大学）；Harbin Institute of Technology（哈尔滨工业大学）；Shanghai Jiao Tong University（上海交通大学）；Institute of Automation（自动化研究所）；Fudan University（复旦大学）；High School Affiliated to Fudan University（复旦大学附属高中）；Renmin University of China（中国人民大学）；University of Washington（华盛顿大学）
While investment funds publicly disclose their objectives in broad terms, their managers optimize for complex combinations of competing goals that go beyond simple risk-return trade-offs. Traditional approaches attempt to model this through multi-objective utility functions, but face fundamental challenges in specification and parameterization. We propose a generative framework that learns latent representations of fund manager strategies without requiring explicit utility specification. Our approach directly models the conditional probability of a fund's portfolio weights, given stock charact…
Skill-RM: Unifying Heterogeneous Evaluation Criteria via Agent Skill
Qwen Large Model Application Team, Alibaba（通义千问大模型应用团队，阿里巴巴）；Sun Yat-sen University（中山大学）；The Chinese University of Hong Kong（香港中文大学）；Peking University（北京大学）；ETH Zürich University of Zurich（苏黎世联邦理工学院）
