基于图神经网络的按键识别
Keystep Recognition using Graph Neural Networks
AI总结：
本文提出GLEVR框架，通过构建图结构有效利用第一人称视频的长期依赖关系，实现细粒度按键识别，优于现有方法。
摘要：
我们将按键识别视为一个节点分类任务，并提出了一种灵活的图学习框架，用于细粒度的按键识别，能够有效利用第一人称视频中的长期依赖关系。我们的方法称为GLEVR，包括构建一个图，其中每个第一人称视频的视频片段对应一个节点。构建的图是稀疏且计算高效的，明显优于现有较大的模型。我们进一步利用第一人称和第二人称视频之间的对齐关系进行训练，以提高对第一人称视频的推理性能，以及添加自动字幕作为额外的模态。我们考虑每个第二人称视频（如果可用）的每个片段或视频字幕作为训练中的额外节点。我们探讨了几种定义这些节点之间连接策略的方法。我们在Ego-Exo4D数据集上进行了广泛的实验，并证明了我们提出的灵活的图基框架在现有方法上显著优于现有方法。
英文摘要：
We pose keystep recognition as a node classification task, and propose a flexible graph-learning framework for fine-grained keystep recognition that is able to effectively leverage long-term dependencies in egocentric videos. Our approach, termed GLEVR, consists of constructing a graph where each video clip of the egocentric video corresponds to a node. The constructed graphs are sparse and computationally efficient, outperforming existing larger models substantially. We further leverage alignment between egocentric and exocentric videos during training for improved inference on egocentric videos, as well as adding automatic captioning as an additional modality. We consider each clip of each exocentric video (if available) or video captions as additional nodes during training. We examine several strategies to define connections across these nodes. We perform extensive experiments on the Ego-Exo4D dataset and show that our proposed flexible graph-based framework notably outperforms existing methods.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
