OST-Bench：评估多模态大语言模型在在线时空场景理解中的能力
OST-Bench: Evaluating the Capabilities of MLLMs in Online Spatio-temporal Scene Understanding
AI总结：
针对现有基准忽视在线时空理解的问题，提出OST-Bench基准，包含1.4k场景和10k问答对，评估多模态大语言模型，发现其复杂时空推理能力不足，并识别出空间推理和记忆检索两大核心挑战。
摘要：
近年来，多模态大语言模型（MLLMs）在整合视觉与语言进行复杂推理方面展现出显著能力。虽然大多数现有基准在离线设置下使用固定的预记录输入来评估模型，我们引入了OST-Bench，这是一个从主动探索场景的智能体视角来评估在线时空理解的基准。在线方面强调处理并推理增量获取的观测数据的必要性，而时空组件则要求将当前视觉输入与历史记忆相结合，以支持动态空间推理。OST-Bench更好地反映了现实世界具身感知的挑战。基于高效的数据收集流程，OST-Bench包含从ScanNet、Matterport3D和ARKitScenes收集的1.4k个场景和10k个问答对。我们在OST-Bench上评估了多个领先的MLLMs，并观察到它们在需要复杂时空推理的任务上表现不足。在在线设置下，随着探索范围的延长和记忆的增长，它们的准确率会下降。通过进一步的实验分析，我们识别了跨模型的常见错误模式，并发现基于线索的复杂空间推理需求和长期记忆检索需求分别沿两个独立轴显著降低了模型性能，这突出了改进在线具身推理必须解决的核心挑战。为了促进该领域的进一步研究和发展，我们的代码、数据集和基准已公开。我们的项目页面是：https://rbler1234.github.io/OSTBench.github.io/
英文摘要：
Recent advances in multimodal large language models (MLLMs) have shown remarkable capabilities in integrating vision and language for complex reasoning. While most existing benchmarks evaluate models under offline settings with a fixed set of pre-recorded inputs, we introduce OST-Bench, a benchmark designed to evaluate Online Spatio-Temporal understanding from the perspective of an agent actively exploring a scene. The Online aspect emphasizes the need to process and reason over incrementally acquired observations, while the Spatio-Temporal component requires integrating current visual inputs with historical memory to support dynamic spatial reasoning. OST-Bench better reflects the challenges of real-world embodied perception. Built on an efficient data collection pipeline, OST-Bench consists of 1.4k scenes and 10k question-answer pairs collected from ScanNet, Matterport3D, and ARKitScenes. We evaluate several leading MLLMs on OST-Bench and observe that they fall short on tasks requiring complex spatio-temporal reasoning. Under the online setting, their accuracy declines as the exploration horizon extends and the memory grows. Through further experimental analysis, we identify common error patterns across models and find that both complex clue-based spatial reasoning demands and long-term memory retrieval requirements significantly drop model performance along two separate axes, highlighting the core challenges that must be addressed to improve online embodied reasoning. To foster further research and development in the field, our codes, dataset, and benchmark are available. Our project page is: https://rbler1234.github.io/OSTBench.github.io/
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
