全景语言模型：超越拼接的全景场景理解
More than the Sum: Panorama-Language Models for Adverse Omni-Scenes
AI总结：
本文提出全景语言模型PLM，通过全景图像实现超越单目视角的综合推理，同时构建包含恶劣全景场景的PanoVQA数据集，开发可插拔的稀疏注意力模块以提升模型处理全景图像的能力。
摘要：
现有视觉-语言模型（VLMs）专为针孔成像设计，通过拼接多个窄视角输入来构建完整的全景场景理解。然而，这种多视角感知忽略了单个全景图像本身固有的整体空间和上下文关系。本文引入全景语言建模（PLM）范式，一种统一的360度视觉-语言推理方法，其效果超越单个针孔模型的总和。此外，我们还提出了PanoVQA，一个大规模全景视觉问答数据集，包含恶劣全景场景，使模型能够在物体遮挡和交通事故等挑战性场景中进行全面推理。为了建立PLM的基础，我们开发了一种可插拔的全景稀疏注意力模块，使现有的基于针孔的VLMs能够处理等距矩形全景图像而无需重新训练。广泛的实验表明，我们的PLM在挑战性全景场景中实现了优越的鲁棒性和整体推理能力，其理解效果超越了其各个窄部分的总和。项目页面：https://github.com/InSAI-Lab/PanoVQA。
英文摘要：
Existing vision-language models (VLMs) are tailored for pinhole imagery, stitching multiple narrow field-of-view inputs to piece together a complete omni-scene understanding. Yet, such multi-view perception overlooks the holistic spatial and contextual relationships that a single panorama inherently preserves. In this work, we introduce the Panorama-Language Modeling (PLM)paradigm, a unified $360^\circ$ vision-language reasoning that is more than the sum of its pinhole counterparts. Besides, we present PanoVQA, a large-scale panoramic VQA dataset that involves adverse omni-scenes, enabling comprehensive reasoning under object occlusions and driving accidents. To establish a foundation for PLM, we develop a plug-and-play panoramic sparse attention module that allows existing pinhole-based VLMs to process equirectangular panoramas without retraining. Extensive experiments demonstrate that our PLM achieves superior robustness and holistic reasoning under challenging omni-scenes, yielding understanding greater than the sum of its narrow parts. Project page: https://github.com/InSAI-Lab/PanoVQA.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
