任务成功并不足够：探究视频-语言模型作为行为批评者以捕捉不期望的智能体行为
Task Success is not Enough: Investigating the Use of Video-Language Models as Behavior Critics for Catching Undesirable Agent Behaviors
AI总结：
本文针对具身智能体中仅以任务成功为验证标准不足的问题，构建了包含不期望行为的基准，系统评估了视频-语言模型作为可扩展行为批评者的能力，并提出了将VLM反馈用于策略迭代优化的实用指南。
摘要：
大规模生成模型已被证明有助于采样有意义的候选解决方案，但它们常常忽视任务约束和用户偏好。当这些模型与外部验证器相结合，并根据验证反馈迭代地或逐步地推导最终解决方案时，其全部能力才能得到更好的发挥。在具身人工智能的语境下，验证通常仅涉及评估指令中指定的目标条件是否已被满足。然而，为了让这些智能体无缝融入日常生活，至关重要的是要考虑超越单纯任务成功的更广泛的约束和偏好（例如，机器人应当小心地抓取面包以避免显著变形）。然而，鉴于机器人任务的范围是无界的，构建类似于用于围棋和定理证明等显式知识任务的脚本化验证器是不可行的。这引出了一个问题：当没有可靠的验证器可用时，我们能否使用近似全知的大型视觉和语言模型（VLMs）作为可扩展的行为批评者，来捕捉视频中不期望的机器人行为？为了回答这个问题，我们首先构建了一个基准，其中包含多种达到目标但行为不期望的机器人策略案例。然后，我们全面评估VLM批评者，以更深入地理解它们的优势和失败模式。基于评估结果，我们提供了关于如何有效利用VLM批评意见的指南，并展示了一种将反馈整合到策略迭代优化过程中的实用方法。数据集和代码库已发布于：https://guansuns.github.io/pages/vlm-critic。
英文摘要：
Large-scale generative models are shown to be useful for sampling meaningful candidate solutions, yet they often overlook task constraints and user preferences. Their full power is better harnessed when the models are coupled with external verifiers and the final solutions are derived iteratively or progressively according to the verification feedback. In the context of embodied AI, verification often solely involves assessing whether goal conditions specified in the instructions have been met. Nonetheless, for these agents to be seamlessly integrated into daily life, it is crucial to account for a broader range of constraints and preferences beyond bare task success (e.g., a robot should grasp bread with care to avoid significant deformations). However, given the unbounded scope of robot tasks, it is infeasible to construct scripted verifiers akin to those used for explicit-knowledge tasks like the game of Go and theorem proving. This begs the question: when no sound verifier is available, can we use large vision and language models (VLMs), which are approximately omniscient, as scalable Behavior Critics to catch undesirable robot behaviors in videos? To answer this, we first construct a benchmark that contains diverse cases of goal-reaching yet undesirable robot policies. Then, we comprehensively evaluate VLM critics to gain a deeper understanding of their strengths and failure modes. Based on the evaluation, we provide guidelines on how to effectively utilize VLM critiques and showcase a practical way to integrate the feedback into an iterative process of policy refinement. The dataset and codebase are released at: https://guansuns.github.io/pages/vlm-critic.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
