分割、嵌入和对齐：将字幕与手语对齐的通用方法
Segment, Embed, and Align: A Universal Recipe for Aligning Subtitles to Signing
AI总结：
提出一种通用框架SEA，利用预训练模型分割视频帧序列为单个手势、嵌入手势片段到与文本共享的潜在空间，并通过轻量动态规划实现高效对齐，在多个手语数据集上达到最先进性能。
摘要：
本文的目标是开发一种通用方法，用于将字幕（即带有对应时间戳的口语文本）与连续手语视频对齐。先前的方法通常依赖于针对特定语言或数据集的端到端训练，这限制了它们的通用性。相比之下，我们的方法Segment, Embed, and Align (SEA)提供了一个适用于多种语言和领域的单一框架。SEA利用两个预训练模型：第一个模型将视频帧序列分割为单个手势，第二个模型将每个手势的视频片段嵌入到与文本共享的潜在空间中。随后，通过轻量级动态规划程序进行对齐，该程序即使在长达一小时的视频中也能在CPU上高效运行，耗时不到一分钟。SEA灵活且能适应各种场景，利用从小型词汇表到大型连续语料库的资源。在四个手语数据集上的实验展示了最先进的对齐性能，突显了SEA在生成高质量并行数据以推动手语处理方面的潜力。SEA的代码和模型已公开提供。
英文摘要：
The goal of this work is to develop a universal approach for aligning subtitles (i.e., spoken language text with corresponding timestamps) to continuous sign language videos. Prior approaches typically rely on end-to-end training tied to a specific language or dataset, which limits their generality. In contrast, our method Segment, Embed, and Align (SEA) provides a single framework that works across multiple languages and domains. SEA leverages two pretrained models: the first to segment a video frame sequence into individual signs and the second to embed the video clip of each sign into a shared latent space with text. Alignment is subsequently performed with a lightweight dynamic programming procedure that runs efficiently on CPUs within a minute, even for hour-long episodes. SEA is flexible and can adapt to a wide range of scenarios, utilizing resources from small lexicons to large continuous corpora. Experiments on four sign language datasets demonstrate state-of-the-art alignment performance, highlighting the potential of SEA to generate high-quality parallel data for advancing sign language processing. SEA's code and models are openly available.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
