基于图的多模态和多视角对齐用于按键识别
Graph-Based Multimodal and Multi-view Alignment for Keystep Recognition
AI总结：
本文提出了一种基于图的多模态和多视角对齐框架，用于提高egocentric视频中按键识别的准确率，通过构建稀疏图结构并利用多模态特征提升性能。
摘要：
egocentric视频捕捉穿戴者视角的场景，导致动态背景、频繁运动和遮挡，给准确的按键识别带来了挑战。我们提出了一种灵活的图学习框架，用于细粒度的按键识别，能够有效利用egocentric视频中的长期依赖关系，并在训练过程中利用egocentric和exocentric视频之间的对齐以提高egocentric视频的推理性能。我们的方法包括构建一个图，其中每个egocentric视频的视频片段对应一个节点。在训练过程中，我们将每个exocentric视频的每个片段（如果可用）作为额外的节点。我们考察了几种策略来定义这些节点之间的连接，并将按键识别视为在构建的图上的节点分类任务。我们在Ego-Exo4D数据集上进行了广泛的实验，并证明了我们提出的灵活的图基框架在准确率上显著优于现有方法，高出12个百分点以上。此外，构建的图是稀疏且计算高效的。我们还展示了一项研究，探讨如何利用多种多模态特征，包括叙述、深度和物体类别标签，在异构图上进行处理，并讨论这些特征对按键识别性能的相应贡献。
英文摘要：
Egocentric videos capture scenes from a wearer's viewpoint, resulting in dynamic backgrounds, frequent motion, and occlusions, posing challenges to accurate keystep recognition. We propose a flexible graph-learning framework for fine-grained keystep recognition that is able to effectively leverage long-term dependencies in egocentric videos, and leverage alignment between egocentric and exocentric videos during training for improved inference on egocentric videos. Our approach consists of constructing a graph where each video clip of the egocentric video corresponds to a node. During training, we consider each clip of each exocentric video (if available) as additional nodes. We examine several strategies to define connections across these nodes and pose keystep recognition as a node classification task on the constructed graphs. We perform extensive experiments on the Ego-Exo4D dataset and show that our proposed flexible graph-based framework notably outperforms existing methods by more than 12 points in accuracy. Furthermore, the constructed graphs are sparse and compute efficient. We also present a study examining on harnessing several multimodal features, including narrations, depth, and object class labels, on a heterogeneous graph and discuss their corresponding contribution to the keystep recognition performance.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
