重定时贝尔曼流：逃离速度自举的不可能三角
Retimed Bellman Flows: Escaping the Impossible Triangle of Velocity Bootstrapping
浏览论文内容
中文总结 AI 辅助
提出重定时贝尔曼流（ReBF），通过动态偏移教师查询时间并解耦噪声生成，构建条件无偏速度目标，解决速度自举的结构困境，在合成MRP和38个离线强化学习任务上显著提升性能。
中文摘要 AI 辅助
流评论家通过连续速度场将高斯噪声传输到贝尔曼端点来学习回报分布。虽然速度自举通过查询后继教师模型稳定了训练，但现有方法面临结构性困境：在直线路径上，不存在无残差的同时间仿射映射能够在保持无偏目标的同时保留高斯初始噪声。为克服这一限制，我们引入了重定时贝尔曼流（ReBF）。ReBF在动态偏移的较早流时间查询教师评论家，使中间学生和教师轨迹对齐。通过将此重定时时钟与全新的、解耦的噪声生成相结合，ReBF构建了一个可证明的条件无偏速度目标，该目标保持贝尔曼不动点并在Wasserstein距离下收缩。实验上，ReBF在合成MRP上将与真实回报分布的$W_1$距离最多减少了$7.7\ imes$，并在38个具有挑战性的OGBench和D4RL离线强化学习任务中优于现有流评论家。
英文摘要
Flow critics learn return distributions by transporting Gaussian noise to Bellman endpoints via continuous velocity fields. While velocity bootstrapping stabilizes training by querying a successor teacher, existing methods face a structural dilemma: on straight paths, no residual-free same-time affine mapping can preserve Gaussian initial noise while maintaining an unbiased target. To overcome this limitation, we introduce Retimed Bellman Flows (ReBF). ReBF queries the teacher critic at a dynamically shifted earlier flow time, aligning intermediate student and teacher trajectories. By combining this retimed clock with fresh, decoupled noise generation, ReBF constructs a provably conditionally unbiased velocity target that preserves the Bellman fixed point and contracts under Wasserstein distances. Empirically, ReBF reduces $W_1$ distance to ground-truth return distributions by up to $7.7\times$ on synthetic MRPs and outperforms existing flow critics across 38 challenging OGBench and D4RL offline reinforcement learning tasks.
