面向点云视频理解的运动显著性互补掩码建模

中文摘要 AI 辅助

点云视频表示学习对三维动态场景理解至关重要。本文提出MoSaiC，一种用于自监督点云视频表示学习的新型运动显著性互补掩码建模框架。MoSaiC包含三个组件：课程式运动显著性掩码（CMSM），其在课程计划引导下将掩码过程导向运动显著的 token；法向流运动（NFM）建模，其将每个token在李代数so(3)中的局部刚性旋转作为显式几何运动目标进行监督；跨视图token一致性预测（CTCP），其在token层面强制两个互补掩码视图间的一致性。这些组件结合使MoSaiC能有效捕获外观与运动动态。在动作识别、时序动作分割、点级语义分割等多个下游任务上开展的大量实验验证了该方法的有效性。

英文摘要

Point cloud video representation learning is crucial for 3D dynamic scene understanding. In this paper, we propose MoSaiC, a novel Motion-Saliency Complementary masked modeling framework for self-supervised point cloud video representation learning. MoSaiC couples three components: Curriculum Motion-Saliency Masking (CMSM), which guides the masking process toward motion-salient tokens under a curriculum schedule; Normal-Flow Motion (NFM) modeling, which supervises the local rigid rotation of each token in the Lie algebra so(3) as an explicit geometric motion target; and Cross-view Token Consistency Prediction (CTCP), which enforces consistency between two complementary masked views at the token level. Together, these components allow MoSaiC to effectively capture both appearance and motion dynamics. Extensive experiments on multiple downstream tasks, including action recognition, temporal action segmentation, and point-level semantic segmentation, demonstrate the effectiveness of our approach.

发表机构

• School of Computer Science and Technology, Xi’an Jiaotong University（西安交通大学计算机科学与技术学院）
• State Key Laboratory of Intelligent Geotechnics and Tunnelling (FSDI)（智能岩土与隧道工程国家重点实验室）
• School of Software Engineering, Xi’an Jiaotong University（西安交通大学软件学院）
• Guangdong Laboratory of Artificial Intelligence and Digital Economy (SZ)（广东省人工智能与数字经济实验室（深圳））
• School of Information and Communication Engineering, Xi’an Jiaotong University（西安交通大学信息与通信工程学院）

机构由 AI 辅助整理，请以论文原文为准。