HMR3D：用于大视觉-语言模型的层次多模态表示以实现3D场景理解
HMR3D: Hierarchical Multimodal Representation for 3D Scene Understanding with Large Vision-Language Model
AI总结：
HMR3D通过层次化多模态表示，结合多视图图像和文本描述，提升3D场景理解的性能。
摘要：
近年来，大视觉-语言模型（VLMs）在3D场景理解方面取得了显著进展。现有的基于VLM的方法通常将3D场景特征对齐到VLM的嵌入空间。然而，这种隐式对齐往往由于3D数据稀缺和3D环境中空间关系的固有复杂性，导致性能不理想。为了解决这些限制，我们提出了一种新的层次多模态表示，用于3D场景推理，通过利用多视图图像和文本描述，显式地对齐到VLM的输入空间。文本描述通过引用检测到的对象的3D坐标来捕捉空间关系，而多视图图像包括俯视视角和四个方向视角（前方、左侧、右侧和后方），确保全面的场景覆盖。此外，我们引入了层次化的特征表示，将图像块级特征聚合为视角级和场景级的表示，使模型能够推理局部和全局场景上下文。在 situated 3D Q&A 和 general 3D Q&A 评估基准上，实验结果证明了我们方法的有效性。
英文摘要：
Recent advances in large vision-language models (VLMs) have shown significant promise for 3D scene understanding. Existing VLM-based approaches typically align 3D scene features with the VLM's embedding space. However, this implicit alignment often yields suboptimal performance due to the scarcity of 3D data and the inherent complexity of spatial relationships in 3D environments. To address these limitations, we propose a novel hierarchical multimodal representation for 3D scene reasoning that explicitly aligns with VLMs at the input space by leveraging both multi-view images and text descriptions. The text descriptions capture spatial relationships by referencing the 3D coordinates of detected objects, while the multi-view images include a top-down perspective and four directional views (forward, left, right, and backward), ensuring comprehensive scene coverage. Additionally, we introduce a hierarchical feature representation that aggregates patch-level image features into view-level and scene-level representations, enabling the model to reason over both local and global scene context. Experimental results on both situated 3D Q&A and general 3D Q&A benchmarks demonstrate the effectiveness of our approach.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
