视觉验证实现推理时引导与自主策略改进
Visual Verification Enables Inference-time Steering and Autonomous Policy Improvement
AI总结：
提出VERITAS框架，利用预训练通用机器人策略作为生成器，结合无梯度视觉验证器在推理时评估动作，实现无需额外训练的推理时策略引导和离线策略改进。
摘要：
部署在现实世界中的机器人应从经验中学习并随时间改进。这需要一个实践并从反馈中学习的机制。在本文中，我们提出VERITAS，一个用于通用机器人策略的生成器-验证器框架，用于推理时策略引导和自我改进。我们使用预训练的通用机器人策略作为“生成器”，并将其与一个无梯度的“视觉验证器”配对，该验证器在推理时评估动作。该框架实现了推理时引导，无需额外训练即可提高策略性能。我们证明，推理时验证在无需额外演示数据训练的情况下，始终优于普通通用策略。此外，我们证明验证后的 rollout 为离线策略改进提供了有效的监督：在验证后的自生成轨迹上微调的策略实现了持续的性能提升。值得注意的是，我们发现使用验证后的 rollout 进行后训练达到了与专家演示相当的效率，同时无需人工干预。我们的结果突出了推理时验证作为一种实用且可扩展的机制，用于在部署期间改进机器人策略。
英文摘要：
Robots deployed in the real world should learn from their experience and improve over time. This requires a mechanism of practicing and learning from feedback. In this paper, we propose VERITAS, a generator-verifier framework for generalist robot policies for inference-time policy steering and self-improvement. We use a pre-trained generalist robot policy as a ``generator'' and pair it with a gradient-free ``visual verifier'' that evaluates actions at inference time. This framework enables inference-time steering that improves policy performance without additional training. We demonstrate that inference-time verification consistently outperforms vanilla generalists without training on additional demonstration data. Additionally, we demonstrate that the verified rollouts provide effective supervision for offline policy improvement: policies fine-tuned on verified self-generated trajectories achieve consistent performance gains. Notably, we find that post-training with verified rollouts achieves comparable efficiency to expert demonstrations, while requiring no human interventions. Our results highlight inference-time verification as a practical and scalable mechanism for improving robotic policies during deployment.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
