从丰富的2D视觉场景中学习4D全景场景图生成
Learning 4D Panoptic Scene Graph Generation from Rich 2D Visual Scene
AI总结：
针对4D全景场景图研究的数据稀缺、词汇表外及流水线方法性能欠佳问题，提出融合3D掩码解码器的4D-LLM、链式SG推理机制与2D到4D迁移学习框架，显著提升了生成性能。
摘要：
最新出现的4D全景场景图（4D Panoptic Scene Graph，4D-PSG）为全面建模动态4D视觉现实世界提供了前所未有的先进表示形式。遗憾的是，当前前沿的4D-PSG研究主要受困于严重的数据稀缺问题，以及由此产生的词汇表外问题；此外，基准生成方法的流水线特性会导致性能欠佳。为应对这些挑战，本文研究了一种全新的4D-PSG生成框架，该框架利用丰富的2D视觉场景标注来增强4D场景学习。首先，我们提出了一种集成3D掩码解码器的4D大语言模型（4D Large Language Model，4D-LLM），用于端到端生成4D-PSG。我们进一步设计了链式场景图（SG）推理机制，以利用大语言模型的开放词汇能力迭代推断出准确且全面的物体与关系标签。最重要的是，我们提出了2D到4D的视觉场景迁移学习框架，其中时空场景跨越策略可将维度不变特征从大量2D SG标注有效迁移到4D场景，有力弥补了4D-PSG的数据稀缺问题。在基准数据上开展的大量实验表明，我们的方法以大幅优势显著优于基线模型，凸显了方法的有效性。
英文摘要：
The latest emerged 4D Panoptic Scene Graph (4D-PSG) provides an advanced-ever representation for comprehensively modeling the dynamic 4D visual real world. Unfortunately, current pioneering 4D-PSG research can primarily suffer from data scarcity issues severely, as well as the resulting out-of-vocabulary problems; also, the pipeline nature of the benchmark generation method can lead to suboptimal performance. To address these challenges, this paper investigates a novel framework for 4D-PSG generation that leverages rich 2D visual scene annotations to enhance 4D scene learning. First, we introduce a 4D Large Language Model (4D-LLM) integrated with a 3D mask decoder for end-to-end generation of 4D-PSG. A chained SG inference mechanism is further designed to exploit LLMs' open-vocabulary capabilities to infer accurate and comprehensive object and relation labels iteratively. Most importantly, we propose a 2D-to-4D visual scene transfer learning framework, where a spatial-temporal scene transcending strategy effectively transfers dimension-invariant features from abundant 2D SG annotations to 4D scenes, effectively compensating for data scarcity in 4D-PSG. Extensive experiments on the benchmark data demonstrate that we strikingly outperform baseline models by a large margin, highlighting the effectiveness of our method.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
