同步即一切：利用无标注同步视频对进行外向到内向的时间动作分割迁移

中文摘要 AI 辅助

我们考虑将最初为外向（固定）相机设计的时间动作分割系统迁移到可穿戴相机捕获视频数据的内向场景的问题。传统的监督方法需要收集和标注一组新的内向视频来适应模型，这既昂贵又耗时。相反，我们提出了一种新颖的方法，利用已有的标注外向视频和一组新的无标注、同步的外向-内向视频对来进行适应，这些视频对无需收集时间动作分割标注。我们使用基于知识蒸馏的方法来实现所提出的方法，并在特征层面和时间动作分割模型层面进行了研究。在Assembly101和EgoExo4D上的实验证明了所提方法相对于经典的无监督域适应和时间对齐方法的有效性。无需任何花哨技巧，我们的最佳模型在性能上与使用标注内向数据训练的监督方法相当，而从未见过任何内向标签，在Assembly101数据集上相比仅用外向数据训练的基线模型，编辑分数提高了+15.99（28.59对12.60）。在类似设置下，我们的方法在具有挑战性的EgoExo4D基准上将编辑分数提高了+3.32。代码可在以下网址获取：https://github.com/fpv-iplab/synchronization-is-all-you-need。

英文摘要

We consider the problem of transferring a temporal action segmentation system initially designed for exocentric (fixed) cameras to an egocentric scenario, where wearable cameras capture video data. The conventional supervised approach requires the collection and labeling of a new set of egocentric videos to adapt the model, which is costly and time-consuming. Instead, we propose a novel methodology which performs the adaptation leveraging existing labeled exocentric videos and a new set of unlabeled, synchronized exocentric-egocentric video pairs, for which temporal action segmentation annotations do not need to be collected. We implement the proposed methodology with an approach based on knowledge distillation, which we investigate both at the feature and Temporal Action Segmentation model level. Experiments on Assembly101 and EgoExo4D demonstrate the effectiveness of the proposed method against classic unsupervised domain adaptation and temporal alignment approaches. Without bells and whistles, our best model performs on par with supervised approaches trained on labeled egocentric data, without ever seeing a single egocentric label, achieving a +15.99 improvement in the edit score (28.59 vs 12.60) on the Assembly101 dataset compared to a baseline model trained solely on exocentric data. In similar settings, our method also improves edit score by +3.32 on the challenging EgoExo4D benchmark. Code is available here: https://github.com/fpv-iplab/synchronization-is-all-you-need.

发表机构

• University of Catania（卡塔尼亚大学）
• Next Vision s.r.l.（Next Vision有限责任公司）
• University of Nottingham（诺丁汉大学）

机构由 AI 辅助整理，请以论文原文为准。