为世界模型评估适应视觉-语言模型
Adapting Vision-Language Models for Evaluating World Models
AI总结：
本文提出UNIVERSE，一种基于视觉-语言模型的视频世界模型评估器，通过适应不同任务格式和数据约束，实现了细粒度、时间敏感的评估，与任务特定检查点性能相当，并在多种环境中验证了其与人类判断的一致性。
摘要：
世界模型——生成模型，能够根据过去观察和行动模拟环境动态——在规划、模拟和具身人工智能中日益突出。然而，评估其回放仍是一个根本性挑战，需要细致的、时间相关的评估行动一致性和语义一致性——这些能力未被现有指标所捕捉。视觉-语言模型（VLMs）已显示出作为生成内容自动评估器的潜力，由于其强大的多模态推理能力。然而，它们在细粒度、时间敏感的评估任务中的应用仍有限，需要有针对性的适应。我们引入了一个评估协议，针对两个识别任务——动作识别和字符识别——每个任务在二元、多选和开放性格式中进行评估。为了支持这一点，我们提出了UNIVERSE（UNIfied Vision-language Evaluator for Rollouts in Simulated Environments），一种基于VLM的视频世界模型回放评估器，在数据和计算约束下进行适应。在我们超过5,154个GPU-天的广泛实验中，我们探索了各种任务格式、上下文长度、采样方法和数据组成下的完整、部分和参数高效适应方法。所得到的统一评估器在任务特定检查点上实现了持平。在七个多样化的环境中进行的人类研究证实了与人类判断的强一致性，确立了UNIVERSE作为视频世界模型的轻量、可适应和语义感知的评估器。
英文摘要：
World models - generative models that simulate environment dynamics conditioned on past observations and actions - are gaining prominence in planning, simulation, and embodied AI. However, evaluating their rollouts remains a fundamental challenge, requiring fine-grained, temporally grounded assessment of action alignment and semantic consistency - capabilities not captured by existing metrics. Vision-Language Models (VLMs) have shown promise as automatic evaluators of generative content due to their strong multimodal reasoning abilities. Yet, their use in fine-grained, temporally sensitive evaluation tasks remains limited and requires targeted adaptation. We introduce an evaluation protocol targeting two recognition tasks - action recognition and character recognition - each assessed across binary, multiple-choice, and open-ended formats. To support this, we present UNIVERSE (UNIfied Vision-language Evaluator for Rollouts in Simulated Environments), a VLM-based evaluator for video world model rollouts adapted under data and compute constraints. In our extensive experiments totaling over 5,154 GPU-days, we explore full, partial, and parameter-efficient adaptation methods across various task formats, context lengths, sampling methods, and data compositions. The resulting unified evaluator achieves parity with task-specific checkpoints. Human studies across seven diverse environments confirm strong alignment with human judgments, establishing UNIVERSE as a lightweight, adaptable, and semantics-aware evaluator for video world models.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
