arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2512.04048cs.CVcs.CLcs.CY
收藏

Stable Signer: 层级化手语生成模型

Stable Signer: Hierarchical Sign Language Generative Model

Rutgers University(罗格斯大学);Nanyang Technological University(南洋理工大学);Georgia Institute of Technology(佐治亚理工学院);University of Wisconsin-Madison(威斯康星大学麦迪逊分校)

Sen Fang, Yalin Feng, Hongbin Zhong, Yanxin Zhang, Dimitris N. Metaxas

AI总结:

本文提出Stable Signer模型,通过层级生成端到端任务提升手语视频生成质量,采用SLUL和SLP-MoE模块实现高效生成。

摘要:

手语生成(SLP)是将复杂的文本输入转换为真实视频的过程。大多数先前的工作集中在Text2Gloss、Gloss2Pose、Pose2Vid阶段,一些则集中在Prompt2Gloss和Text2Avatar阶段。然而,由于在这些阶段中的文本转换、姿态生成以及姿态渲染为真实人类视频的不准确性,导致逐步积累的错误,使该领域进展缓慢。因此,在本文中,我们简化并优化了传统冗余的结构,重新定义SLP任务为一个层级生成的端到端任务,仅包含文本理解(Prompt2Gloss、Text2Gloss)和Pose2Vid。通过我们提出的新的手语理解链接器称为SLUL来执行文本理解,并通过名为SLP-MoE的手势渲染专家块生成手势,以端到端生成高质量且多风格的手语视频。SLUL通过新开发的语义感知手势掩码损失(SAGM Loss)进行训练。其性能相比当前最先进的生成方法提高了48.6%。

英文摘要:

Sign Language Production (SLP) is the process of converting the complex input text into a real video. Most previous works focused on the Text2Gloss, Gloss2Pose, Pose2Vid stages, and some concentrated on Prompt2Gloss and Text2Avatar stages. However, this field has made slow progress due to the inaccuracy of text conversion, pose generation, and the rendering of poses into real human videos in these stages, resulting in gradually accumulating errors. Therefore, in this paper, we streamline the traditional redundant structure, simplify and optimize the task objective, and design a new sign language generative model called Stable Signer. It redefines the SLP task as a hierarchical generation end-to-end task that only includes text understanding (Prompt2Gloss, Text2Gloss) and Pose2Vid, and executes text understanding through our proposed new Sign Language Understanding Linker called SLUL, and generates hand gestures through the named SLP-MoE hand gesture rendering expert block to end-to-end generate high-quality and multi-style sign language videos. SLUL is trained using the newly developed Semantic-Aware Gloss Masking Loss (SAGM Loss). Its performance has improved by 48.6% compared to the current SOTA generation methods.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑