发表机构
University of Oxford; Microsoft; Simon Fraser University（牛津大学; 微软公司; 西蒙弗雷泽大学）
机构由 AI 辅助整理，请以论文原文为准。AI 中文总结
研究针对现有3D-LLMs不能处理多房间场景问题，提出拓扑感知3D-LLM即CAIRN。其将Transformer注意力与场景层次对齐，通过多种方式增强模型能力。在新基准CAIRN-MR上实验，CAIRN在多房间任务中大幅超越前人，单房间任务也具竞争力。
AI 中文摘要
现有的3D场景基础大语言模型（3D-LLMs）专注于回答基于简化单房间3D场景的问题，缺乏对包含多个相互连接房间和多样物体类别的现实家庭环境进行推理的能力。我们引入了CAIRN，一种用于多房间3D场景理解的拓扑感知3D-LLM。CAIRN将Transformer注意力与场景层次结构对齐，使模型明确了解对象级关系和房间级连接性。它通过图神经网络用房间局部关系上下文丰富对象令牌，引入用于房间级抽象的学习房间令牌，并应用带有几何偏差的分层注意力掩码根据场景拓扑路由信息。CAIRN是在CAIRN-MR上开发的，CAIRN-MR是我们在HM3D上引入的用于多房间3D场景理解的基准，涵盖基础、字幕和从房间内感知到跨房间推理逐步评估的四个问答任务。实验表明，CAIRN在所有CAIRN-MR任务上大幅优于先前的3D-LLMs，同时在五个单房间基准测试中保持竞争力。
英文摘要
Existing 3D scene-grounded Large Language Models (3D-LLMs) focus on answering questions grounded in simplified single-room 3D scenes, lacking the ability to reason over real-world household environments containing multiple interconnected rooms and diverse object categories. We introduce CAIRN, a topology-aware 3D-LLM for multi-room 3D scene understanding. CAIRN aligns transformer attention with scene hierarchy, giving the model explicit awareness of object-level relations and room-level connectivity. It enriches object tokens with room-local relational context via a graph neural network, introduces learned room tokens for room-level abstraction, and applies a hierarchical attention mask with geometric bias to route information according to scene topology. CAIRN is developed on CAIRN-MR, a benchmark we introduce on HM3D for multi-room 3D scene understanding, covering grounding, captioning, and four question-answering tasks that progressively evaluate from intra-room perception to cross-room reasoning. Experiments show that CAIRN outperforms prior 3D-LLMs by a large margin across all CAIRN-MR tasks while remaining competitive on five single-room benchmarks.
CommentsProject Page: https://oceansdepp.github.io/cairn_web/
