arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2603.29281cs.CVcs.AIcs.RO
收藏

PRISM:一个多视角多能力零售视频数据集用于具身视觉-语言模型

PRISM: A Multi-View Multi-Capability Retail Video Dataset for Embodied Vision-Language Models

DreamVu

Amirreza Rouhi, Parikshit Sakurikar, Satya Sai Reddy, Narsimha Menga, Anirudh Govil, Sri Harsha Chittajallu, Rajat Aggarwal, Anoop Namboodiri, Sashi Reddi

AI总结:

PRISM是首个针对真实世界零售环境的多视角多能力视频数据集,通过领域特定的SFT训练提升具身视觉-语言模型的感知能力与空间理解。

摘要:

在最先进的物理AI模型的通用视觉理解与结构化现实世界部署环境的专用感知需求之间存在关键差距。我们提出了PRISM,一个包含27万样本的多视角视频监督微调(SFT)数据集,用于真实世界零售环境中的具身视觉-语言模型(VLMs)。PRISM的动机源于一个简单观察——物理AI系统失败并非因为视觉识别能力差,而是因为它们对空间、物理动态和具身动作的理解不足,无法可靠地在现实世界中运行。为此,PRISM基于一个新颖的三维知识本体,涵盖空间知识、时间与物理知识以及具身动作知识。它覆盖了四个评估维度下的20多个能力探针——具身推理(ER)、常识(CS)、空间感知(SP)和直观物理(IP),并且据我们所知,PRISM是首个在单一现实世界部署领域内实例化所有三个知识维度的数据集。该数据集从五个超市地点的中心视觉、外视觉和360°视角采集数据,并包含开放性、链式思维和多项选择监督。在4fps下,PRISM涵盖约1.18亿视频帧和73亿个token,使其成为最大的领域特定视频SFT数据集之一。在PRISM上进行微调可将所有20多个探针的错误率降低66.6%,在具身动作理解方面准确率提升36.4%。我们的结果表明,基于本体结构的领域特定SFT可以显著增强具身VLMs在现实世界中的表现。PRISM数据集和更多细节可在https://dreamvu.ai/prism获取。

英文摘要:

A critical gap exists between the general-purpose visual understanding of state-of-the-art physical AI models and the specialized perceptual demands of structured real-world deployment environments. We present PRISM, a 270K-sample multi-view video supervised fine-tuning (SFT) corpus for embodied vision-language-models (VLMs) in real-world retail environments. PRISM is motivated by a simple observation - physical AI systems fail not because of poor visual recognition, but because they do not understand space, physical dynamics and embodied action well enough to operate reliably in the world. To this end, PRISM is grounded in a novel three-dimensional knowledge ontology that spans spatial knowledge, temporal and physical knowledge, and embodied action knowledge. It covers 20+ capability probes across four evaluation dimensions - Embodied Reasoning (ER), Common Sense (CS), Spatial Perception (SP), and Intuitive Physics (IP), and to our knowledge, PRISM is the first dataset to instantiate all three knowledge dimensions within a single real-world deployment domain. The corpus captures data from egocentric, exocentric and 360° viewpoints across five supermarket locations and includes open-ended, chain-of-thought, and multiple-choice supervision. At 4 fps, PRISM spans approximately 11.8M video frames and approximately 730M tokens, placing it among the largest domain-specific video SFT corpora. Fine-tuning on PRISM reduces the error rate across all 20+ probes by 66.6% over the pre-trained baseline, with significant gains in embodied action understanding where the accuracy improves by 36.4%. Our results suggest that ontology-structured, domain specific SFT can meaningfully strengthen embodied VLMs for real-world settings. The PRISM dataset and more details are available at https://dreamvu.ai/prism

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑