4DPC$^2$hat: 面向动态点云理解的失败感知自举学习
4DPC$^2$hat: Towards Dynamic Point Cloud Understanding with Failure-Aware Bootstrapping
AI总结：
提出首个针对动态点云理解的多模态大语言模型4DPC$^2$hat，通过构建大规模跨模态数据集4DPC$^2$hat-200K和引入Mamba增强的时间推理模块及失败感知自举学习策略，显著提升了动作理解与时间推理能力。
摘要：
点云提供了3D对象的紧凑且富有表现力的表示，最近已被集成到多模态大语言模型（MLLMs）中。然而，现有方法主要关注静态对象，而理解动态点云序列仍基本未被探索。这一限制主要是由于缺乏大规模跨模态数据集以及在时空上下文中建模运动的难度。为弥补这一差距，我们提出了4DPC$^2$hat，这是首个专为动态点云理解设计的MLLM。为此，我们通过一个精心设计的两阶段流程构建了大规模跨模态数据集4DPC$^2$hat-200K，该流程包括拓扑一致的4D点构建和两级标注。该数据集包含超过44K个动态对象序列、700K个点云帧和200K个精心策划的问答对，支持关于计数、时间关系、动作、空间关系和外观的查询。在该框架的核心，我们引入了一个Mamba增强的时间推理MLLM，以捕捉点云序列中的长程依赖和动态模式。此外，我们提出了一种失败感知的自举学习策略，该策略迭代地识别模型缺陷并生成有针对性的问答监督，以持续增强相应的推理能力。大量实验表明，与现有模型相比，我们的4DPC$^2$hat显著提高了动作理解和时间推理能力，为4D动态点云理解奠定了坚实基础。
英文摘要：
Point clouds provide a compact and expressive representation of 3D objects, and have recently been integrated into multimodal large language models (MLLMs). However, existing methods primarily focus on static objects, while understanding dynamic point cloud sequences remains largely unexplored. This limitation is mainly caused by the lack of large-scale cross-modal datasets and the difficulty of modeling motions in spatio-temporal contexts. To bridge this gap, we present 4DPC$^2$hat, the first MLLM tailored for dynamic point cloud understanding. To this end, we construct a large-scale cross-modal dataset 4DPC$^2$hat-200K via a meticulous two-stage pipeline consisting of topology-consistent 4D point construction and two-level captioning. The dataset contains over 44K dynamic object sequences, 700K point cloud frames, and 200K curated question-answer (QA) pairs, supporting inquiries about counting, temporal relationship, action, spatial relationship, and appearance. At the core of the framework, we introduce a Mamba-enhanced temporal reasoning MLLM to capture long-range dependencies and dynamic patterns among a point cloud sequence. Furthermore, we propose a failure-aware bootstrapping learning strategy that iteratively identifies model deficiencies and generates targeted QA supervision to continuously strengthen corresponding reasoning capabilities. Extensive experiments demonstrate that our 4DPC$^2$hat significantly improves action understanding and temporal reasoning compared with existing models, establishing a strong foundation for 4D dynamic point cloud understanding.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
