首页/论文/2002.05522

arXiv 2002.055222020-02cs.LGcs.AIstat.ML

BRPO: Batch Residual Policy Optimization

Google Research（谷歌研究院）；University of Michigan（密歇根大学）

Sungryull Sohn, Yinlam Chow, Jayden Ooi, Ofir Nachum, Honglak Lee, Ed Chi, Craig Boutilier

英文摘要：

In batch reinforcement learning (RL), one often constrains a learned policy to be close to the behavior (data-generating) policy, e.g., by constraining the learned action distribution to differ from the behavior policy by some maximum degree that is the same at each state. This can cause batch RL to be overly conservative, unable to exploit large policy changes at frequently-visited, high-confidence states without risking poor performance at sparsely-visited states. To remedy this, we propose residual policies, where the allowable deviation of the learned policy is state-action-dependent. We derive a new for RL method, BRPO, which learns both the policy and allowable deviation that jointly maximize a lower bound on policy performance. We show that BRPO achieves the state-of-the-art performance in a number of tasks.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。

阅读原文

阅读 PDF ↗

arXiv 页面 ↗HTML 原文 ↗

本文内容

英文摘要相关论文

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。

继续探索

相关论文

Bayesian Residual Policy Optimization: Scalable Bayesian Reinforcement Learning with Clairvoyant Experts

University of Washington（华盛顿大学）

Informed and robust decision making in the face of uncertainty is critical for robots that perform physical tasks alongside people. We formulate this as Bayesian Reinforcement Learning over latent Markov Decision Processes (MDPs). While Bayes-optimality is theoretically the gold standard, existing algorithms do not scale well to continuous state and action spaces. Our proposal builds on the following insight: in the absence of uncertainty, each latent MDP is easier to solve. We first obtain an ensemble of experts, one for each latent MDP, and fuse their advice to compute a baseline policy. Nex…

2020-02-07cs.LGcs.RO

用于不确定性感知强化学习的广义高斯时间差分误差

Generalized Gaussian Temporal Difference Error for Uncertainty-aware Reinforcement Learning

Qraft Technologies Data Team（Qraft Technologies数据团队）

研究用于不确定性感知强化学习的广义高斯时间差分误差，引入基于广义高斯分布的状态条件形状头，用数值修改的GGD损失作在线替代，构建加权启发式方法并提出BIEV正则化，形状感知变体在部分基准测试中表现更优。

2026-07-27cs.AIcs.LGmath.PR

Analysis on Gradient Propagation in Batch Normalized Residual Networks

University of Southern California（南加州大学）

We conduct mathematical analysis on the effect of batch normalization (BN) on gradient backpropogation in residual network training, which is believed to play a critical role in addressing the gradient vanishing/explosion problem, in this work. By analyzing the mean and variance behavior of the input and the gradient in the forward and backward passes through the BN and residual branches, respectively, we show that they work together to confine the gradient variance to a certain range across residual blocks in backpropagation. As a result, the gradient vanishing/explosion problem is avoided. W…

2018-12-02cs.LGstat.ML

面向社交导航的真实世界学习的增量残差强化学习

Incremental Residual Reinforcement Learning Toward Real-World Learning for Social Navigation

School of Engineering, Kyushu University（九州大学工学系）；Faculty of Information Science and Electrical Engineering, Kyushu University（九州大学信息科学与电子工学系）；School of Information Science and Electrical Engineering, Kyushu University（九州大学信息科学与电子工学系）

提出增量残差强化学习（IRRL），结合增量学习与残差强化学习，在无回放缓冲区的情况下实现与基于回放缓冲区方法相当的性能，并通过真实世界实验验证机器人适应新环境的能力。

2026-06-25cs.AIcs.RO

基于零阶残差跟踪的部分信息随机优化

Partially Informed Stochastic Optimization via Zeroth-Order Residual Tracking

EPFL（洛桑联邦理工学院）；Machine Learning and Optimization Laboratory (MLO)（机器学习与优化实验室）

针对仅能获取部分梯度的优化问题，提出PISO方法，结合可用梯度与残差跟踪，在非凸优化中获领先复杂度，在多任务中优于多数基线方法。

2026-10-12cs.LG

Batch Normalization Biases Residual Blocks Towards the Identity Function in Deep Networks

DeepMind（深度思维）

Batch normalization dramatically increases the largest trainable depth of residual networks, and this benefit has been crucial to the empirical success of deep residual networks on a wide range of benchmarks. We show that this key benefit arises because, at initialization, batch normalization downscales the residual branch relative to the skip connection, by a normalizing factor on the order of the square root of the network depth. This ensures that, early in training, the function computed by normalized residual blocks in deep networks is close to the identity function (on average). We use th…

2020-12-09cs.CVcs.LGstat.ML

更多相关论文收起更多论文

SAW: 面向大语言模型多目标强化学习的阶段感知动态加权

SAW: Stage-Aware Dynamic Weighting for Multi-Objective Reinforcement Learning in Large Language Models

Institute of Computing Technology, Chinese Academy of Sciences（中国科学院计算技术研究所）；University of Electronic Science and Technology of China（电子科技大学）

针对多目标强化学习中奖励学习异步性问题，提出轻量级动态加权机制SAW，利用变异系数实时调整各目标贡献，在GRPO和GDPO框架下提升训练效率和最终性能。

2026-06-05cs.AIcs.LG

让稀疏奖励发挥作用：面向多奖励强化学习的密度感知奖励聚合

Make Sparse Rewards Count: Density-Aware Reward Aggregation for Multi-Reward RL

University of Chinese Academy of Sciences（中国科学院大学）；University of Minnesota Twin Cities（明尼苏达大学双城分校）；University of Wisconsin–Madison（威斯康星大学麦迪逊分校）；Stony Brook University（纽约州立大学石溪分校）；Dalian University of Technology（大连理工大学）；Beijing Foreign Studies University（北京外国语大学）；Southeast University（东南大学）；Kuaishou Technology（快手科技）

针对多奖励强化学习中奖励稀疏导致的学习不均衡问题，提出密度感知奖励聚合（DARA），通过逆平方根密度校正加权奖励信号，加速工具调用与数学推理任务的学习，减少训练步数。

2026-10-02cs.CLcs.LG

SALT: 当更多 rollout 在基于组的策略优化中无益时如何使其发挥作用

SALT: When More Rollouts Don't Help in Group-Based Policy Optimization and How to Make Them Matter

Bilibili Inc.（哔哩哔哩公司）；Fudan University（复旦大学）；Zhejiang University（浙江大学）

针对 GRPO 风格组归一化中增加 rollout 数量导致梯度抵消的问题，提出 SALT 组件，通过子空间自适应重加权组相对更新系数，改善更新几何并提升性能。

2026-06-04cs.LG

How to Use Dropout Correctly on Residual Networks with Batch Normalization

Pohang University of Science and Technology（浦项科技大学）

For the stable optimization of deep neural networks, regularization methods such as dropout and batch normalization have been used in various tasks. Nevertheless, the correct position to apply dropout has rarely been discussed, and different positions have been employed depending on the practitioners. In this study, we investigate the correct position to apply dropout. We demonstrate that for a residual network with batch normalization, applying dropout at certain positions increases the performance, whereas applying dropout at other positions decreases the performance. Based on theoretical an…

2023-02-13cs.CVcs.LG

一种用于无需批归一化的ResNet有效训练的残差块鲁棒初始化方法

A Robust Initialization of Residual Blocks for Effective ResNet Training without Batch Normalization

Flair Tech（Flair Tech公司）

针对批归一化带来的实际问题，该研究通过微调残差块输出与跳跃连接的求和操作实现正确初始化，成功训练无归一化的类ResNet网络，在多个主流图像数据集上取得了有竞争力的结果。

2022-10-12cs.LG

Dynamical Isometry for Residual Networks

CISPA Helmholtz Center for Information Security（CISPA亥姆霍兹信息安全中心）

The training success, training speed and generalization ability of neural networks rely crucially on the choice of random parameter initialization. It has been shown for multiple architectures that initial dynamical isometry is particularly advantageous. Known initialization schemes for residual blocks, however, miss this property and suffer from degrading separability of different inputs for increasing depth and instability without Batch Normalization or lack feature diversity. We propose a random initialization scheme, RISOTTO, that achieves perfect dynamical isometry for residual networks w…

2022-10-05cs.LG