反事实传输流用于离线保守轨迹细化
Counterfactual Transport Flows for Offline Conservative Trajectory Refinement
AI总结：
提出反事实传输流框架，通过检索高反馈轨迹构建局部偏好对，实现离线决策的保守轨迹细化，在D4RL基准上提升历史回报表现。
摘要：
离线强化学习提供了一条仅从记录数据中改进策略的路径，使用历史回报或其他可测量的结果作为世界反馈。一个关键困难是在不超出离线数据支持范围的情况下改进观察到的行为。我们提出了反事实传输流，这是一个由世界反馈引导的、用于离线决策的源条件轨迹细化框架。给定一个低反馈候选轨迹，我们通过在潜在轨迹空间中检索具有更高任务特定反馈的邻近轨迹，从离线数据中构建局部偏好对，并将其用作保守细化的弱监督。该框架学习实例特定的细化方向：在推理时，细化强度参数控制候选轨迹被传输的距离，从而在保留原始行为和施加更强改进之间实现权衡。在包括AntMaze和MuJoCo任务的D4RL基准上的实验表明，我们的方法从作为世界反馈的历史回报中改进行为，同时提供可解释的轨迹级细化路径。
英文摘要：
Offline reinforcement learning (RL) offers a path to policy improvement from logged data alone, using historical returns or other measurable outcomes as world feedback. A key difficulty is improving observed behavior without extrapolating beyond what the offline data supports. We propose \emph{counterfactual transport flows}, a source-conditioned trajectory refinement framework for offline decision-making guided by world feedback. Given a low-feedback candidate trajectory, we construct local preference pairs from offline data by retrieving nearby trajectories in latent trajectory space with higher task-specific feedback, and use them as weak supervision for conservative refinement. The framework learns instance-specific refinement directions: at inference time, a refinement strength parameter controls how far the candidate trajectory is transported, enabling a trade-off between preserving the original behavior and applying stronger improvement. Experiments on D4RL benchmarks, including AntMaze and MuJoCo tasks, show that our method improves behavior from historical returns as world feedback, while providing interpretable trajectory-level refinement paths.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
