发表机构
MIT; MIT Lincoln Laboratory（麻省理工学院; 麻省理工学院林肯实验室）
机构由 AI 辅助整理，请以论文原文为准。AI 中文总结
LASER通过潜空间伴随匹配实现熵正则化的离线RL，避免时间反向传播，在40个OGBench任务上以固定超参数达到最先进性能。
AI 中文摘要
虽然离线强化学习（RL）能够从静态数据集进行策略优化而无需昂贵的在线交互，但其仍受限于执行分布外（OOD）动作的风险。近期方法通过流匹配学习行为克隆策略，然后在其受约束的潜空间内执行RL来缓解这一问题。然而，天真地优化潜策略容易导致策略崩溃为脆弱的模式或利用学习到的评论家的尖锐伪影。在这项工作中，我们发现熵正则化对于解决潜空间RL中的这些挑战至关重要。我们引入了LASER，一种新颖的离线RL算法，应用潜空间伴随匹配来实现具有表达性流策略的熵正则化潜空间RL，同时避免通过时间的反向传播。通过在40个具有不同数据集质量的具有挑战性的OGBench任务上的全面实验，我们展示了LASER达到了最先进的性能。值得注意的是，LASER在所有任务中使用固定的方法特定超参数，并超越了评估的基线（包括那些具有任务和数据集特定调优的基线），这突显了LASER的稳健适用性。项目网站：此HTTPS URL。
英文摘要
While offline reinforcement learning (RL) enables policy optimization from static datasets without costly online interaction, it remains bottlenecked by the risk of executing out-of-distribution (OOD) actions. Recent approaches mitigate this by learning a behavior-cloning policy through flow matching and then performing RL within its constrained latent space. However, naively optimizing the latent policy can easily cause the policy to collapse into a brittle mode or exploit sharp artifacts of the learned critic. In this work, we find that entropy regularization is essential in latent-space RL for addressing these challenges. We introduce LASER, a novel offline RL algorithm that applies latent-space adjoint matching to achieve entropy-regularized latent-space RL with expressive flow policies while avoiding backpropagation through time. Through comprehensive experiments on 40 challenging OGBench tasks with varying dataset qualities, we show that LASER achieves state-of-the-art performance. Notably, LASER uses fixed method-specific hyperparameters across all tasks and outperforms the evaluated baselines, including those with task- and dataset-specific tuning, which highlights the robust applicability of LASER. Project website: https://mit-realm.github.io/laser/.
Comments29 pages, 16 figures. Accepted at NeurIPS 2026
