视觉、手语与言说：用于从社交媒体获取并整理手语数据的视觉-语言模型辅助流水线
Seeing, Signing, and Saying: A Vision-Language Model-Assisted Pipeline for Sign Language Data Acquisition and Curation from Social Media
AI总结：
针对现有手语翻译数据集规模小、成本高的问题，提出首个基于视觉语言模型的自动标注过滤流水线，可从社交媒体多语言手语视频中获取整理数据，建立相关模型评估基准，支撑可扩展弱监督预训练。
摘要：
现有大多数手语翻译（SLT）数据集规模有限、缺乏多语言覆盖，且因依赖专家标注与受控录制环境，整理成本高昂。近期，视觉语言模型（VLM）已展现出作为评估器与实时助手的强大能力。尽管有这些进展，其在手语数据集获取场景中的潜力仍未被挖掘。为填补这一空白，我们提出首个利用VLM减少对手工工作的依赖、同时保障数据质量的自动标注与过滤框架。我们的方法被应用于涵盖8种手语的TikTok视频，以及已整理好的德国手语YouTube-SL-25数据集以开展额外评估。我们基于VLM的流水线包含人脸可见性检测、手语活动识别、视频内容文本提取，以及验证视频与文本对齐性的判断步骤，实现了通用的过滤、标注与验证流程。利用生成的语料库TikTok-SL-8，我们在过滤后的德国手语与美国手语数据集上评估了两款现成SLT模型的性能，目标是建立基准并评估近期模型在自动提取的、存在轻微噪声的数据上的鲁棒性。我们的工作为SLT提供了可扩展的弱监督预训练支持，并推动了从社交媒体获取数据的进程。
英文摘要：
Most existing sign language translation (SLT) datasets are limited in scale, lack multilingual coverage, and are costly to curate due to their reliance on expert annotation and controlled recording setup. Recently, Vision Language Models (VLMs) have demonstrated strong capabilities as evaluators and real-time assistants. Despite these advancements, their potential remains untapped in the context of sign language dataset acquisition. To bridge this gap, we introduce the first automated annotation and filtering framework that utilizes VLMs to reduce reliance on manual effort while preserving data quality. Our method is applied to TikTok videos across eight sign languages and to the already curated YouTube-SL-25 dataset in German Sign Language for the purpose of additional evaluation. Our VLM-based pipeline includes a face visibility detection, a sign activity recognition, a text extraction from video content, and a judgment step to validate alignment between video and text, implementing generic filtering, annotation and validation steps. Using the resulting corpus, TikTok-SL-8, we assess the performance of two off-the-shelf SLT models on our filtered dataset for German and American Sign Languages, with the goal of establishing baselines and evaluating the robustness of recent models on automatically extracted, slightly noisy data. Our work enables scalable, weakly supervised pretraining for SLT and facilitates data acquisition from social media.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
