arXiv 2605.082022026-05-06cs.LGcs.AI

超越惩罚：基于扩散的分布外检测与选择性正则化在离线强化学习中

Beyond Penalization: Diffusion-based Out-of-Distribution Detection and Selective Regularization in Offline Reinforcement Learning

School of Computer Science and Technology, Tongji University, Shanghai, China（同济大学计算机科学与技术学院，上海，中国）；MOE Key Lab of Embedded System and Service Computing, Tongji University, Shanghai, China（教育部嵌入式系统与服务计算重点实验室，同济大学，上海，中国）；Shanghai Innovation Institute（上海创新研究院）

Qingjun Wang, Hongtu Zhou, Hang Yu, Junqiao Zhao, Yanping Zhao, Chen Ye, Ziqiao Wang, Guang Chen

备注：: 10 pages, 5 figures. Accepted to ICLR 2026

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

本文提出DOSER框架，通过扩散模型捕捉行为策略和状态分布，利用单步去噪重构误差检测分布外动作，并通过评估预测转移区分有益与有害动作，提供理论保证和实验验证。

摘要：

离线强化学习面临过度估计分布外（OOD）动作价值的挑战。现有方法通过惩罚未见样本缓解此问题，但无法准确识别OOD动作且可能抑制有益探索。本文提出DOSER（基于扩散的OOD检测与选择性正则化），通过训练两个扩散模型捕捉行为策略和状态分布，利用单步去噪重构误差作为可靠OOD指标。在策略优化中，通过评估预测转移区分有益与有害OOD动作，选择性抑制风险动作并鼓励探索高潜力动作。理论上证明DOSER是γ-收缩映射，因此具有唯一固定点和有界价值估计。进一步提供模型近似和OOD检测误差下的渐进行为保证。在广泛离线RL基准测试中，DOSER在子最优数据集上表现优于现有方法。

英文摘要：

Offline reinforcement learning (RL) faces a critical challenge of overestimating the value of out-of-distribution (OOD) actions. Existing methods mitigate this issue by penalizing unseen samples, yet they fail to accurately identify OOD actions and may suppress beneficial exploration beyond the behavioral support. Although several methods have been proposed to differentiate OOD samples with distinct properties, they typically rely on restrictive assumptions about the data distribution and remain limited in discrimination ability. To address this problem, we propose DOSER (Diffusion-based OOD Detection and Selective Regularization), a novel framework that goes beyond uniform penalization. DOSER trains two diffusion models to capture the behavior policy and state distribution, using single-step denoising reconstruction error as a reliable OOD indicator. During policy optimization, it further distinguishes between beneficial and detrimental OOD actions by evaluating predicted transitions, selectively suppressing risky actions while encouraging exploration of high-potential ones. Theoretically, we prove that DOSER is a $γ$-contraction and therefore admits a unique fixed point with bounded value estimates. We further provide an asymptotic performance guarantee relative to the optimal policy under model approximation and OOD detection errors. Across extensive offline RL benchmarks, DOSER consistently attains superior performance to prior methods, especially on suboptimal datasets.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。