通过折扣存活公式评估操纵策略的离线策略评估
Offline Policy Evaluation for Manipulation Policies via Discounted Liveness Formulation
AI总结：
本文提出基于存活Bellman算子的离线策略评估框架，解决稀疏奖励下任务完成问题，通过保守固定点值函数降低截断偏差，实验证明在折叠任务中优于传统基线方法。
摘要：
策略评估是机器人策略开发和部署流程中的基本组件。在现代操纵系统中，这一问题尤为具有挑战性：奖励通常稀疏，评估回放的任务进展往往非单调，因为策略表现出恢复行为，而评估回放本质上是有限长度的。这种有限长度引入了截断偏差，破坏了标准方法所依赖的无限时间 horizon 假设。在本文中，我们提出了一种基于存活Bellman算子的离线策略评估框架，以稀疏奖励为基础。我们的公式将策略评估解释为任务完成问题，并产生一个保守的固定点值函数，该函数对有限时间截断具有鲁棒性。我们分析了所提出算子的理论性质，包括收缩保证，并展示了其如何编码任务进展同时缓解截断偏差。我们在两个模拟操纵任务上使用了视觉-语言-动作模型和扩散策略，并在使用人类演示的折叠任务上进行了评估。实验证明，我们的方法更准确地反映了任务进展，并显著减少了截断偏差，优于经典基线方法如TD(0)和蒙特卡洛策略评估。
英文摘要：
Policy evaluation is a fundamental component of the development and deployment pipeline for robotic policies. In modern manipulation systems, this problem is particularly challenging: rewards are often sparse, task progression of evaluation rollouts are often non-monotonic as the policies exhibit recovery behaviors, and evaluation rollouts are necessarily of finite length. This finite length introduces truncation bias, breaking the infinite-horizon assumptions underlying standard methods relying on Bellman equations/principle of optimality. In this work, we propose a framework for offline policy evaluation from sparse rewards based on a liveness-based Bellman operator. Our formulation interprets policy evaluation as a task-completion problem and yields a conservative fixed-point value function that is robust to finite-horizon truncation. We analyze the theoretical properties of the proposed operator, including contraction guarantees, and show how it encodes task progression while mitigating truncation bias. We evaluate our method on two simulated manipulation tasks using both a Vision-Language-Action model and a diffusion policy, and a cloth folding task using human demonstrations. Empirical results demonstrate that our approach more accurately reflects task progress and substantially reduces truncation bias, outperforming classical baselines such as TD(0) and Monte Carlo policy evaluation.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
