ODI-Bench: MLLMs能否理解沉浸式全向环境？
ODI-Bench: Can MLLMs Understand Immersive Omnidirectional Environments?
AI总结：
ODI-Bench旨在评估多模态大语言模型对全向图像沉浸环境的理解能力，通过定制基准和Omni-CoT方法提升模型表现。
摘要：
全向图像（ODI）提供360x180度的全景视角，被广泛应用于VR、AR和具身智能应用。尽管多模态大语言模型（MLLMs）在传统2D图像和视频理解基准上表现出色，但其对通过ODI捕获的沉浸式环境的理解能力仍鲜有探索。为此，我们首先提出了ODI-Bench，一个专门用于全向图像理解的新型综合基准。ODI-Bench包含2000张高质量全向图像和超过4000个手动标注的问答（QA）对，覆盖10个细粒度任务，包括通用级和空间级的ODI理解。进行了大量实验，以在封闭式和开放式设置下对20个代表性的MLLMs进行基准测试，包括专有和开源模型。实验结果表明，当前的MLLMs仍然难以捕捉ODI提供的沉浸式上下文。为此，我们进一步引入了Omni-CoT，一种无需训练的方法，通过跨文本信息和视觉线索的推理显著提升MLLMs在全向环境中的理解能力。该基准和代码将在https://github.com/ylylyl-sjtu/ODI-Bench发布。
英文摘要：
Omnidirectional images (ODIs) provide full 360x180 view which are widely adopted in VR, AR and embodied intelligence applications. While multi-modal large language models (MLLMs) have demonstrated remarkable performance on conventional 2D image and video understanding benchmarks, their ability to comprehend the immersive environments captured by ODIs remains largely unexplored. To address this gap, we first present ODI-Bench, a novel comprehensive benchmark specifically designed for omnidirectional image understanding. ODI-Bench contains 2,000 high-quality omnidirectional images and over 4,000 manually annotated question-answering (QA) pairs across 10 fine-grained tasks, covering both general-level and spatial-level ODI understanding. Extensive experiments are conducted to benchmark 20 representative MLLMs, including proprietary and open-source models, under both close-ended and open-ended settings. Experimental results reveal that current MLLMs still struggle to capture the immersive context provided by ODIs. To this end, we further introduce Omni-CoT, a training-free method which significantly enhances MLLMs' comprehension ability in the omnidirectional environment through chain-of-thought reasoning across both textual information and visual cues. Both the benchmark and the code will be released at https://github.com/ylylyl-sjtu/ODI-Bench.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
