arXiv 2412.012922025-02-02cs.CV

LSceneLLM：利用自适应视觉偏好增强大型3D场景理解

LSceneLLM: Enhancing Large 3D Scene Understanding Using Adaptive Visual Preferences

South China University of Technology（华南理工大学）；Tencent Robotics X（腾讯Robotics X实验室）；Northeastern University（东北大学）；UMass Amherst（马萨诸塞大学阿默斯特分校）；MIT-IBM Watson AI Lab（麻省理工学院-IBM沃森人工智能实验室）；Pazhou Laboratory（琶洲实验室）；Sichuan University（四川大学）

Hongyan Zhi, Peihao Chen, Junyan Li, Shuailei Ma, Xinyu Sun, Tianhang Xiang, Yinjie Lei, Mingkui Tan, Chuang Gan

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

提出自适应框架LSceneLLM，利用LLM视觉偏好识别任务相关区域并结合场景放大器捕捉细粒度细节，同时引入跨房间基准XR-Scene，显著提升了大型3D场景理解能力。

摘要：

3D视觉-语言模型（3D-VLMs）的研究正受到越来越多的关注，这对于在3D场景中开发具身AI（如视觉导航和具身问答）至关重要。由于视觉特征密度高，尤其是在大型3D场景中，准确定位与任务相关的视觉信息极具挑战性。现有工作试图分割所有对象，并将其特征作为场景表示。然而，这些与任务无关的对象特征包含大量冗余信息，且缺失任务相关区域的细节。为了解决这些问题，我们提出了LSceneLLM，这是一个自适应框架，通过利用LLM对不同任务的视觉偏好来自动识别任务相关区域，随后使用一个即插即用的场景放大器模块来捕捉聚焦区域的细粒度细节。具体而言，密集令牌选择器检查LLM的注意力图，以识别对指令输入的视觉偏好。然后，它放大聚焦区域的细粒度细节。利用自适应自注意力模块来融合粗粒度和所选的细粒度视觉信息。为了全面评估3D-VLMs的大型场景理解能力，我们进一步引入了跨房间理解基准XR-Scene，该基准包含一系列大型场景理解任务，包括XR-QA、XR-EmbodiedPlanning和XR-SceneCaption。实验表明，我们的方法在大型场景理解和现有场景理解基准上均超越了现有方法。将我们的场景放大器模块引入现有3D-VLMs也带来了显著提升。

英文摘要：

Research on 3D Vision-Language Models (3D-VLMs) is gaining increasing attention, which is crucial for developing embodied AI within 3D scenes, such as visual navigation and embodied question answering. Due to the high density of visual features, especially in large 3D scenes, accurately locating task-relevant visual information is challenging. Existing works attempt to segment all objects and consider their features as scene representations. However, these task-agnostic object features include much redundant information and missing details for the task-relevant area. To tackle these problems, we propose LSceneLLM, an adaptive framework that automatically identifies task-relevant areas by leveraging LLM's visual preference for different tasks, followed by a plug-and-play scene magnifier module to capture fine-grained details in focused areas. Specifically, a dense token selector examines the attention map of LLM to identify visual preferences for the instruction input. It then magnifies fine-grained details of the focusing area. An adaptive self-attention module is leveraged to fuse the coarse-grained and selected fine-grained visual information. To comprehensively evaluate the large scene understanding ability of 3D-VLMs, we further introduce a cross-room understanding benchmark, XR-Scene, which contains a series of large scene understanding tasks including XR-QA, XR-EmbodiedPlanning, and XR-SceneCaption. Experiments show that our method surpasses existing methods on both large scene understanding and existing scene understanding benchmarks. Plunging our scene magnifier module into the existing 3D-VLMs also brings significant improvement.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。