寻找标志：识别连续视频中的孤立标志
Looking for the Signs: Identifying Isolated Sign Instances in Continuous Video Footage
摘要：
在本文中，我们专注于一次性符号识别的任务，即给定一个孤立的符号（查询）的例子，我们要确定这个符号是否/在哪里出现在一个连续的，共同表达的手语视频（目标）。为了实现这一目标，我们提出了一个基于变压器的网络，称为Signaling。我们采用3D卷积神经网络（CNN）从视频片段中提取时空表示。为了解决查询和目标视频之间的时间尺度差异，我们使用不同的帧级步幅从单个视频片段构造多个查询。自注意力被应用于这些查询片段以模拟连续的尺度空间。我们还利用目标视频上的另一个自我注意模块来学习序列中的上下文。最后，使用相互注意来匹配时间尺度以在目标序列内定位查询。大量的实验表明，该方法不仅可以可靠地识别孤立的迹象，在连续的视频，无论签名者的外观，但也可以推广到不同的手语。通过利用注意力机制和自适应特性，我们的模型在具有挑战性的基准数据集上实现了高达96%的准确率，并且显著优于其他方法。
英文摘要：
In this paper, we focus on the task of one-shot sign spotting, i.e. given an example of an isolated sign (query), we want to identify whether/where this sign appears in a continuous, co-articulated sign language video (target). To achieve this goal, we propose a transformer-based network, called SignLookup. We employ 3D Convolutional Neural Networks (CNNs) to extract spatio-temporal representations from video clips. To solve the temporal scale discrepancies between the query and the target videos, we construct multiple queries from a single video clip using different frame-level strides. Self-attention is applied across these query clips to simulate a continuous scale space. We also utilize another self-attention module on the target video to learn the contextual within the sequence. Finally a mutual-attention is used to match the temporal scales to localize the query within the target sequence. Extensive experiments demonstrate that the proposed approach can not only reliably identify isolated signs in continuous videos, regardless of the signers' appearance, but can also generalize to different sign languages. By taking advantage of the attention mechanism and the adaptive features, our model achieves state-of-the-art performance on the sign spotting task with accuracy as high as 96% on challenging benchmark datasets and significantly outperforming other approaches.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
