arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2412.01800cs.CV
收藏

PhysGame:揭示游戏视频中的物理常识违背现象

PhysGame: Uncovering Physical Commonsense Violations in Gameplay Videos

Mohamed bin Zayed University of Artificial Intelligence(穆罕默德·本·扎耶德人工智能大学);Peking University(北京大学);Alibaba Group(阿里巴巴集团);University of Waterloo(滑铁卢大学);University of Toronto(多伦多大学);Sun Yat-sen University(中山大学)

Meng Cao, Haoran Tang, Haoze Zhao, Hangyu Guo, Jiaheng Liu, Ge Zhang, Ruyang Liu, Qiang Sun, Ian Reid, Xiaodan Liang

AI总结:

本文构建 PhysGame 基准评估游戏视频中的物理常识违背,并通过 PhysInstruct、PhysDPO 训练物理知识增强的 PhysVLM,使其在物理与通用视频理解任务上达到先进水平。

摘要:

近期,基于视频的大型语言模型(Video LLM)取得进展,已涌现出对动态视觉内容进行推理和解释的多样化能力。其中,游戏视频作为一种独特数据源尤为突出,这类视频常包含违背物理常识的画面故障。该特征使其成为评估 Video LLM 中尚未被充分探索的物理常识理解能力的有效基准。本文提出 PhysGame,这是一个用于评估游戏视频中物理常识违背现象的开创性基准。PhysGame 包含 880 个与故障相关的视频,涵盖四个基础领域(即力学、运动学、光学和材料属性),并涉及 12 种不同的物理常识。通过广泛评估多种最先进的 Video LLM,我们发现当前开源 Video LLM 的性能显著落后于专有模型。为弥补这一差距,我们构建了包含 140,057 个问答对的指令微调数据集 PhysInstruct,以促进物理常识学习。此外,我们还提出了包含 34,358 个训练对的偏好优化数据集 PhysDPO,其中不偏好回答分别基于误导性标题(即元信息攻击)、更少帧(即时序攻击)和更低空间分辨率(即空间攻击)生成。基于这一系列数据集,我们提出 PhysVLM,这是一种物理知识增强的 Video LLM。在面向物理的基准 PhysGame 以及通用视频理解基准上的大量实验证明,PhysVLM 具有最先进性能。

英文摘要:

Recent advancements in video-based large language models (Video LLMs) have witnessed the emergence of diverse capabilities to reason and interpret dynamic visual content. Among them, gameplay videos stand out as a distinctive data source, often containing glitches that defy physics commonsense. This characteristic renders them an effective benchmark for assessing the under-explored capability of physical commonsense understanding in video LLMs. In this paper, we propose PhysGame as a pioneering benchmark to evaluate physical commonsense violations in gameplay videos. PhysGame comprises 880 videos associated with glitches spanning four fundamental domains (i.e., mechanics, kinematics, optics, and material properties) and across 12 distinct physical commonsense. Through extensively evaluating various state-ofthe-art video LLMs, our findings reveal that the performance of current open-source video LLMs significantly lags behind that of proprietary counterparts. To bridge this gap, we curate an instruction tuning dataset PhysInstruct with 140,057 question-answering pairs to facilitate physical commonsense learning. In addition, we also propose a preference optimization dataset PhysDPO with 34,358 training pairs, where the dis-preferred responses are generated conditioned on misleading titles (i.e., meta information hacking), fewer frames (i.e., temporal hacking) and lower spatial resolutions (i.e., spatial hacking). Based on the suite of datasets, we propose PhysVLM as a physical knowledge-enhanced video LLM. Extensive experiments on both physical-oriented benchmark PhysGame and general video understanding benchmarks demonstrate the state-ofthe-art performance of PhysVLM.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑