自举你的视角：掩码自我-外部建模用于细粒度视角不变视频表示
Bootstrap Your Own Views: Masked Ego-Exo Modeling for Fine-grained View-invariant Video Representations
AI总结：
本文提出BYOV方法，通过掩码自我-外部建模同时学习因果时间动态和跨视角对齐，从不成对视频中提取细粒度视角不变表示，在四个下游任务中显著优于现有方法。
摘要：
从自我中心（第一人称，自我）和外部中心（第三人称，外部）视频中学习视角不变的表示，是跨多个视角泛化视频理解系统的一种有前景的方法。然而，由于自我视角和外部视角在视角、运动模式和上下文上的显著差异，这一领域尚未得到充分探索。在本文中，我们提出了一种新颖的掩码自我-外部建模方法，该方法同时促进因果时间动态和跨视角对齐，称为“自举你的视角”（BYOV），用于从不成对的自我-外部视频中学习细粒度的视角不变视频表示。我们强调了捕捉人类动作组合性质作为稳健跨视角理解基础的重要性。具体而言，设计了自我视角掩码和跨视角掩码预测，以同时学习视角不变且强大的表示。实验结果表明，我们的BYOV在四个下游自我-外部视频任务中显著优于现有方法，在所有指标上均取得了显著提升。代码可在 https://github.com/park-jungin/byov 获取。
英文摘要：
View-invariant representation learning from egocentric (first-person, ego) and exocentric (third-person, exo) videos is a promising approach toward generalizing video understanding systems across multiple viewpoints. However, this area has been underexplored due to the substantial differences in perspective, motion patterns, and context between ego and exo views. In this paper, we propose a novel masked ego-exo modeling that promotes both causal temporal dynamics and cross-view alignment, called Bootstrap Your Own Views (BYOV), for fine-grained view-invariant video representation learning from unpaired ego-exo videos. We highlight the importance of capturing the compositional nature of human actions as a basis for robust cross-view understanding. Specifically, self-view masking and cross-view masking predictions are designed to learn view-invariant and powerful representations concurrently. Experimental results demonstrate that our BYOV significantly surpasses existing approaches with notable gains across all metrics in four downstream ego-exo video tasks. The code is available at https://github.com/park-jungin/byov.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
