通过时间对齐从非配对Ego-Exo视频中学习细粒度视角不变表示
Learning Fine-grained View-Invariant Representations from Unpaired Ego-Exo Videos via Temporal Alignment
AI总结：
针对自我中心与外部中心视角差异大且缺乏配对同步数据的问题，提出自监督方法AE2，通过以对象为中心的编码器和基于对比的时间对齐目标，从非配对视频中学习细粒度视角不变表示，并在四个数据集上取得优异效果。
摘要：
人类活动的自我中心视角（egocentric）和外部中心视角（exocentric）看起来截然不同，然而建立它们之间联系的不变表示对于机器人和增强现实中的许多潜在应用至关重要。先前的工作仅限于从配对的同步视角中学习视角不变特征。我们放宽了这一强数据假设，提出通过在时间上对齐自我中心视频和外部中心视频，来学习对视角不变的细粒度动作特征，即使这些视频并非同时或在同一环境中捕获。为此，我们提出了AE2，这是一种自监督嵌入方法，包含两个关键设计：（1）一个以对象为中心的编码器，明确关注与手和活动对象相对应的区域；（2）一个基于对比的对齐目标，利用时间反转帧作为负样本。为了进行评估，我们在ego-exo背景下建立了一个细粒度视频理解基准，包含四个数据集——包括我们收集的一个自我视角网球正手击球数据集，以及我们为每个数据集标注的密集逐帧标签。在这四个数据集上，我们的AE2方法在常规和跨视角设置下的多种细粒度下游任务中，均大幅优于先前的工作。
英文摘要：
The egocentric and exocentric viewpoints of a human activity look dramatically different, yet invariant representations to link them are essential for many potential applications in robotics and augmented reality. Prior work is limited to learning view-invariant features from paired synchronized viewpoints. We relax that strong data assumption and propose to learn fine-grained action features that are invariant to the viewpoints by aligning egocentric and exocentric videos in time, even when not captured simultaneously or in the same environment. To this end, we propose AE2, a self-supervised embedding approach with two key designs: (1) an object-centric encoder that explicitly focuses on regions corresponding to hands and active objects; and (2) a contrastive-based alignment objective that leverages temporally reversed frames as negative samples. For evaluation, we establish a benchmark for fine-grained video understanding in the ego-exo context, comprising four datasets -- including an ego tennis forehand dataset we collected, along with dense per-frame labels we annotated for each dataset. On the four datasets, our AE2 method strongly outperforms prior work in a variety of fine-grained downstream tasks, both in regular and cross-view settings.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
