Cautious Policy Programming: Exploiting KL Regularization in Monotonic Policy Improvement for Reinforcement Learning

谨慎乐观：一种在一般博弈中实现近常数遗憾的元算法

Cautious Optimism: A Meta-Algorithm for Near-Constant Regret in General Games

本文提出谨慎乐观元算法框架COFTRL，可自适应调控FTRL学习节奏，在一般博弈自博弈场景达近最优对数遗憾，对抗场景保平方根遗憾，无需单调步长且对动作空间维度依赖指数级优于既往方法。

通过谨慎乐观主义在一般博弈中实现无悔学习的更快收敛速率

Faster Rates for No-Regret Learning in General Games via Cautious Optimism

本文提出首个非耦合学习算法，结合经典OMWU与自适应非单调学习率，在多人一般和博弈中实现 $O(n \log^2 d \log T)$ 的每玩家遗憾值，指数级改善了对动作数量的依赖并降低了对迭代次数的依赖。

Learning from Sparse Data by Exploiting Monotonicity Constraints

When training data is sparse, more domain knowledge must be incorporated into the learning algorithm in order to reduce the effective size of the hypothesis space. This paper builds on previous work in which knowledge about qualitative monotonicities was formally represented and incorporated into learning algorithms (e.g., Clark & Matwin's work with the CN2 rule learning algorithm). We show how to interpret knowledge of qualitative influences, and in particular of monotonicities, as constraints on probability distributions, and to incorporate this knowledge into Bayesian network learning algor…

利用近似对称性实现高效的多智能体强化学习

Exploiting Approximate Symmetry for Efficient Multi-Agent Reinforcement Learning

针对现实场景中缺乏精确对称性和已知模型的问题，提出将有限玩家非对称博弈扩展为诱导MFG的方法，利用近似对称性实现无需显式模型的对称化学习，并证明了其近似Nash策略的收敛性与样本复杂度。

Model-Augmented Actor-Critic: Backpropagating through Paths

Current model-based reinforcement learning approaches use the model simply as a learned black-box simulator to augment the data for policy optimization or value function learning. In this paper, we show how to make more effective use of the model by exploiting its differentiability. We construct a policy optimization algorithm that uses the pathwise derivative of the learned model and policy across future timesteps. Instabilities of learning across many timesteps are prevented by using a terminal value function, learning the policy in an actor-critic fashion. Furthermore, we present a derivati…

Sample-efficient Iterative Lower Bound Optimization of Deep Reactive Policies for Planning in Continuous MDPs

Recent advances in deep learning have enabled optimization of deep reactive policies (DRPs) for continuous MDP planning by encoding a parametric policy as a deep neural network and exploiting automatic differentiation in an end-to-end model-based gradient descent framework. This approach has proven effective for optimizing DRPs in nonlinear continuous MDPs, but it requires a large number of sampled trajectories to learn effectively and can suffer from high variance in solution quality. In this work, we revisit the overall model-based DRP objective and instead take a minorization-maximization p…