首页/论文/2001.06808

arXiv 2001.068082020-01-31cs.LGstat.ML

Discriminator Soft Actor Critic without Extrinsic Rewards

Kanazawa University（金泽大学）

Daichi Nishio, Daiki Kuyoshi, Toi Tsuneda, Satoshi Yamane

英文摘要：

It is difficult to be able to imitate well in unknown states from a small amount of expert data and sampling data. Supervised learning methods such as Behavioral Cloning do not require sampling data, but usually suffer from distribution shift. The methods based on reinforcement learning, such as inverse reinforcement learning and generative adversarial imitation learning (GAIL), can learn from only a few expert data. However, they often need to interact with the environment. Soft Q imitation learning addressed the problems, and it was shown that it could learn efficiently by combining Behavioral Cloning and soft Q-learning with constant rewards. In order to make this algorithm more robust to distribution shift, we propose Discriminator Soft Actor Critic (DSAC). It uses a reward function based on adversarial inverse reinforcement learning instead of constant rewards. We evaluated it on PyBullet environments with only four expert trajectories.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。

阅读原文

阅读 PDF ↗

arXiv 页面 ↗HTML 原文 ↗

本文内容

英文摘要相关论文

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。

继续探索

相关论文

带判别器的外部奖励软Q模仿学习

Extrinsicaly Rewarded Soft Q Imitation Learning with Discriminator

Kanazawa University（金泽大学）

针对少量数据下模仿学习的分布偏移与交互成本问题，提出结合对抗逆强化学习奖励的DSQIL算法，在SQIL基础上提升鲁棒性与效率，并在MuJoCo环境完成评估。

2024-01-30cs.AIcs.LG

Discriminator-Actor-Critic: Addressing Sample Inefficiency and Reward Bias in Adversarial Imitation Learning

Courant Institute of Mathematical Sciences, New York University（纽约大学柯朗数学科学研究所）；Google Brain（谷歌大脑）

We identify two issues with the family of algorithms based on the Adversarial Imitation Learning framework. The first problem is implicit bias present in the reward functions used in these algorithms. While these biases might work well for some environments, they can also lead to sub-optimal behavior in others. Secondly, even though these algorithms can learn from few expert demonstrations, they require a prohibitively large number of interactions with the environment in order to imitate the expert for many real-world applications. In order to address these issues, we propose a new algorithm c…

2018-10-15cs.LGstat.ML

Variational Discriminator Bottleneck: Improving Imitation Learning, Inverse RL, and GANs by Constraining Information Flow

University of California, Berkeley（加州大学伯克利分校）

Adversarial learning methods have been proposed for a wide range of applications, but the training of adversarial models can be notoriously unstable. Effectively balancing the performance of the generator and discriminator is critical, since a discriminator that achieves very high accuracy will produce relatively uninformative gradients. In this work, we propose a simple and general technique to constrain information flow in the discriminator by means of an information bottleneck. By enforcing a constraint on the mutual information between the observations and the discriminator's internal repr…

2020-08-25cs.LGstat.ML

ACtuAL: Actor-Critic Under Adversarial Learning

MILA, Université de Montréal（蒙特利尔大学 MILA）；Polytechnique Montréal（蒙特利尔理工学院）；McGill University（麦吉尔大学）

Generative Adversarial Networks (GANs) are a powerful framework for deep generative modeling. Posed as a two-player minimax problem, GANs are typically trained end-to-end on real-valued data and can be used to train a generator of high-dimensional and realistic images. However, a major limitation of GANs is that training relies on passing gradients from the discriminator through the generator via back-propagation. This makes it fundamentally difficult to train GANs with discrete data, as generation in this case typically involves a non-differentiable function. These difficulties extend to the …

2017-11-13cs.LGstat.ML

Adversarial Advantage Actor-Critic Model for Task-Completion Dialogue Policy Learning

Microsoft Research（微软研究院）；The Chinese University of Hong Kong（香港中文大学）；National Taiwan University（国立台湾大学）

This paper presents a new method --- adversarial advantage actor-critic (Adversarial A2C), which significantly improves the efficiency of dialogue policy learning in task-completion dialogue systems. Inspired by generative adversarial networks (GAN), we train a discriminator to differentiate responses/actions generated by dialogue agents from responses/actions by experts. Then, we incorporate the discriminator as another critic into the advantage actor-critic (A2C) framework, to encourage the dialogue agent to explore state-action within the regions where the agent takes actions similar to tho…

2018-02-08cs.AIcs.CLcs.LG

利用外在行为好奇心实现策略行为多样化

Diversifying Policy Behaviors with Extrinsic Behavioral Curiosity

School of Data Science, The Chinese University of Hong Kong, Shenzhen, China（数据科学学院，香港中文大学（深圳））；CFAR, Agency for Science, Technology and Research, Singapore（CFAR，新加坡科技研究局）；College of Computing and Data Science, Nanyang Technological University (NTU), Singapore（计算与数据科学学院，南洋理工大学（NTU））

针对模仿学习局限于单一策略的问题，提出质量多样性逆强化学习框架QD-IRL，并引入外在行为好奇心EBC机制，通过外部评论家提供的好奇心奖励引导智能体探索多样化行为，显著提升了多种机器人移动任务的性能。

2025-08-14cs.AIcs.LG

更多相关论文收起更多论文

Rethinking Multidimensional Discriminator Output for Generative Adversarial Networks

Salesforce（赛富时）；Amazon（亚马逊）；Florida State University（佛罗里达州立大学）

The study of multidimensional discriminator (critic) output for Generative Adversarial Networks has been underexplored in the literature. In this paper, we generalize the Wasserstein GAN framework to take advantage of multidimensional critic output and explore its properties. We also introduce a square-root velocity transformation (SRVT) block which favors training in the multidimensional setting. Proofs of properties are based on our proposed maximal p-centrality discrepancy, which is bounded above by p-Wasserstein distance and fits the Wasserstein GAN framework with multidimensional critic o…

2022-07-14cs.LGstat.ML

Curiosity-driven Exploration by Self-supervised Prediction

University of California, Berkeley（加州大学伯克利分校）

In many real-world scenarios, rewards extrinsic to the agent are extremely sparse, or absent altogether. In such cases, curiosity can serve as an intrinsic reward signal to enable the agent to explore its environment and learn skills that might be useful later in its life. We formulate curiosity as the error in an agent's ability to predict the consequence of its own actions in a visual feature space learned by a self-supervised inverse dynamics model. Our formulation scales to high-dimensional continuous state spaces like images, bypasses the difficulties of directly predicting pixels, and, c…

2017-05-15cs.AIcs.CVcs.LG

Automatic Reward Design via Learning Motivation-Consistent Intrinsic Rewards

NetEase Fuxi AI Lab（网易伏羲人工智能实验室）

Reward design is a critical part of the application of reinforcement learning, the performance of which strongly depends on how well the reward signal frames the goal of the designer and how well the signal assesses progress in reaching that goal. In many cases, the extrinsic rewards provided by the environment (e.g., win or loss of a game) are very sparse and make it difficult to train agents directly. Researchers usually assist the learning of agents by adding some auxiliary rewards in practice. However, designing auxiliary rewards is often turned to a trial-and-error search for reward setti…

2022-07-29cs.AIcs.LG

Learning more skills through optimistic exploration

DeepMind（深度思维）

Unsupervised skill learning objectives (Gregor et al., 2016, Eysenbach et al., 2018) allow agents to learn rich repertoires of behavior in the absence of extrinsic rewards. They work by simultaneously training a policy to produce distinguishable latent-conditioned trajectories, and a discriminator to evaluate distinguishability by trying to infer latents from trajectories. The hope is for the agent to explore and master the environment by encouraging each skill (latent) to reliably reach different states. However, an inherent exploration problem lingers: when a novel state is actually encounte…

2022-05-12cs.AIcs.LGstat.ML

Deep Intrinsically Motivated Continuous Actor-Critic for Efficient Robotic Visuomotor Skill Learning

University of Hamburg（汉堡大学）

In this paper, we present a new intrinsically motivated actor-critic algorithm for learning continuous motor skills directly from raw visual input. Our neural architecture is composed of a critic and an actor network. Both networks receive the hidden representation of a deep convolutional autoencoder which is trained to reconstruct the visual input, while the centre-most hidden representation is also optimized to estimate the state value. Separately, an ensemble of predictive world models generates, based on its learning progress, an intrinsic reward signal which is combined with the extrinsic…

2019-02-18cs.AIcs.LGcs.RO

通过InfoMax终止批评器学习多样化选项

Learning Diverse Options via InfoMax Termination Critic

Okinawa Institute of Science and Technology（冲绳科学技术大学院大学）；The University of Tokyo（东京大学）

本文提出InfoMax终止批评器（IMTC）算法，通过最大化选项与状态转移之间的互信息来学习选项终止条件，从而提升选项多样性，并在复杂操控任务中验证了其可复用性和快速适应能力。

2023-05-31cs.AIcs.LG

深度学习用于时间序列分类与外部回归：当前综述

Deep Learning for Time Series Classification and Extrinsic Regression: A Current Survey

Monash University（莫纳什大学）；IRIMAS（IRIMAS研究所）；University of Haute-Alsace（上阿尔萨斯大学）

本文综述了深度学习在时间序列分类与外部回归中的最新进展，涵盖网络架构、训练方法、挑战与机遇，并总结人类活动识别和卫星对地观测两大应用。

2023-12-19cs.AIcs.CVcs.LG

TriHead-GAN: 一种具有三头判别器的生成对抗网络用于碳排放时间序列生成

TriHead-GAN: A Generative Adversarial Network with Triple-Head Discriminator for Carbon Emission Time Series Generation

SanMuGuo

针对城市级高频碳排放数据稀缺问题，提出TriHead-GAN，通过三头判别器联合监督分布真实性、跨变量依赖和步态平滑性，在多个数据集上优于主流基线并提升下游预测精度。

2026-09-01cs.LG

错误信念图谱：通过注意力模式追踪内在与外在幻觉

The Map of Misbelief: Tracing Intrinsic and Extrinsic Hallucinations Through Attention Patterns

针对LLMs幻觉检测中采样策略成本高、忽视幻觉类型区分的问题，提出区分内外在幻觉的评估框架，结合基于注意力的不确定性量化算法及新聚合策略，提升检测性能与可解释性，为幻觉检测提供新方向。

2025-11-13cs.AIcs.CLcs.LG

Task-Relevant Adversarial Imitation Learning

Jagiellonian University（雅盖隆大学）；DeepMind（深度思维）

We show that a critical vulnerability in adversarial imitation is the tendency of discriminator networks to learn spurious associations between visual features and expert labels. When the discriminator focuses on task-irrelevant features, it does not provide an informative reward signal, leading to poor task performance. We analyze this problem in detail and propose a solution that outperforms standard Generative Adversarial Imitation Learning (GAIL). Our proposed method, Task-Relevant Adversarial Imitation Learning (TRAIL), uses constrained discriminator optimization to learn informative rewa…

2020-11-12cs.AIcs.LGcs.RO

CHAIN：通过 Lipschitz 连续性约束归一化增强数据高效 GAN 的泛化能力

CHAIN: Enhancing Generalization in Data-Efficient GANs via lipsCHitz continuity constrAIned Normalization

The Australian National University（澳大利亚国立大学）；Data61；CSIRO（联邦科学与工业研究组织）

针对数据受限 GAN 中 BN 导致梯度爆炸及判别器过拟合的问题，本文提出 CHAIN 归一化，通过零均值正则化、Lipschitz 约束和自适应特征插值提升稳定性与泛化，并在多个基准上达到 SOTA。

2025-03-15cs.CVcs.LG