arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2605.01720cs.CVcs.AIcs.CL
收藏

SignVerse-2M:包含55+种手语的两百万片段姿态原生数据集

SignVerse-2M: A Two-Million-Clip Pose-Native Universe of 55+ Sign Languages

Rutgers University(罗格斯大学);Georgia Institute of Technology(佐治亚理工学院);Wisconsin-Madison(威斯康星大学麦迪逊分校)

Sen Fang, Hongbin Zhong, Yanxin Zhang, Liwei Che, Dimitris N. Metaxas

AI总结:

本文提出SignVerse-2M,这是一个包含55+种手语、约200万片段的大规模多语言姿态原生手语数据集,适配现代姿态驱动的生成与识别范式,兼具真实场景适配性,可支撑多语言手语姿态建模与评估。

摘要:

现有大规模手语资源通常仅在原始视频-文本对齐层面提供监督,且多在实验室环境中生成。这类资源对语义理解固然重要,但无法直接为开放世界识别与翻译,或现代姿态驱动的手语视频生成框架提供统一接口:1. 基于RGB的预训练识别模型严重依赖录制时的固定背景或衣物条件,在开放世界场景中的鲁棒性弱于与风格无关的姿态处理模型。2. 近期的姿态引导图像/视频生成模型大多使用DWPose这类统一关键点表示作为控制接口。当前手语领域仍缺乏既能直接适配这种现代姿态原生范式,又能面向真实世界开放场景的数据资源。我们提出SignVerse-2M,这是一个用于手语姿态建模与评估的大规模多语言姿态原生数据集。该数据集构建自公开的多语言手语视频资源,采用DWPose作为统一预处理流程,将原始视频转换为可直接用于建模的2D姿态序列,最终得到包含约200万片段、覆盖55种以上手语的整合语料库。与许多实验室数据集不同,该资源保留了真实世界视频的录制条件与说话人多样性,同时通过统一的姿态表示减少了外观变化。为实现这一目标,我们进一步提供了数据构建流程、任务定义以及简单的SignDW Transformer基线,证明了该资源用于多语言姿态空间建模的可行性及其与现代姿态驱动流程的兼容性,同时讨论了它可支持的评估主张以及当前的局限性。

英文摘要:

Large-scale sign language resources usually pair RGB videos with text and support recognition, translation, and generation. However, RGB frames contain backgrounds, clothing, and other appearance information unrelated to signing motion. Representing the videos as whole-body keypoints such as DWPose removes much of this variation, produces a compact description of body, hand, and facial movement, and matches the input used by recent pose-guided video models. Existing datasets do not provide this representation at scale for in-the-wild sign language videos. We present SignVerse-2M, a multilingual dataset containing approximately two million pose clips from more than 55 sign languages. We process publicly available sign language videos with a shared DWPose pipeline, representing body, hand, and facial motion in the same keypoint format across all clips. The source videos include diverse signers, viewpoints, and recording environments. We also release the processing pipeline, task definitions, and SignDW Transformer baselines. Experiments show that the data supports multilingual text-to-pose modeling and that the generated poses can directly drive several recent pose-conditioned rendering systems.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑