UrbanVideo-Bench: 基于城市空间视频数据评估视觉-语言模型的具身智能
UrbanVideo-Bench: Benchmarking Vision-Language Models on Embodied Intelligence with Video Data in Urban Spaces
AI总结：
UrbanVideo-Bench通过城市空间视频数据评估视频大语言模型的具身智能，揭示其在城市环境中感知、推理和导航能力的局限性，并验证了Sim-to-Real迁移的潜力。
摘要：
大规模多模态模型展现出显著的智能，但其在开放-ended城市3D空间中运动时的具身认知能力仍待探索。我们引入了一个基准测试，评估视频大语言模型（Video-LLMs）是否能自然处理连续的第一人称视觉观察，如人类般实现回忆、感知、推理和导航。我们手动操控无人机，从真实城市和模拟环境收集3D具身运动视频数据，产生1.5k视频片段。然后设计了一个管道生成5.2k多选题。对17种广泛使用的Video-LLMs进行评估，揭示了当前在城市具身认知方面的局限性。相关性分析提供了不同任务之间的关系洞察，显示因果推理与回忆、感知和导航有较强的关联，而反事实和联想推理能力与其他任务的关联性较低。我们还通过微调验证了在城市具身中的Sim-to-Real迁移潜力。
英文摘要：
Large multimodal models exhibit remarkable intelligence, yet their embodied cognitive abilities during motion in open-ended urban 3D space remain to be explored. We introduce a benchmark to evaluate whether video-large language models (Video-LLMs) can naturally process continuous first-person visual observations like humans, enabling recall, perception, reasoning, and navigation. We have manually control drones to collect 3D embodied motion video data from real-world cities and simulated environments, resulting in 1.5k video clips. Then we design a pipeline to generate 5.2k multiple-choice questions. Evaluations of 17 widely-used Video-LLMs reveal current limitations in urban embodied cognition. Correlation analysis provides insight into the relationships between different tasks, showing that causal reasoning has a strong correlation with recall, perception, and navigation, while the abilities for counterfactual and associative reasoning exhibit lower correlation with other tasks. We also validate the potential for Sim-to-Real transfer in urban embodiment through fine-tuning.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
