SignMimic：通过人体形状无关的姿态迁移指导实现鲁棒高质量手语动作生成
SignMimic: Robust High-Quality Sign Language Motion Generation via Human-Shape-Oblivious Pose Transfer Guidance
- New York University Abu Dhabi（纽约大学阿布扎比分校）
- ChatSign Technology（ChatSign 科技）
机构由 AI 辅助整理，请以论文原文为准。
中文总结 AI 辅助
SignMimic通过分解刚体规范化、非刚体适应和姿态补全，利用SE(3)和NIF2D等技术，实现高质量手语视频生成，在多个数据集上达到最先进性能。
中文摘要 AI 辅助
我们研究手语视频模仿的挑战：给定一个驱动视频和单个参考帧，合成一个视频，其中目标手语者再现源动作，同时保持身份和语言形式。先前的流程将刚体运动、非刚体变形和视角相关补全纠缠在一个整体生成器中，导致手形漂移和时空不稳定性。我们提出SignMimic，它（i）应用基于TNet的模型研究SE(3)刚体规范化以稳定全局姿态，（ii）在规范空间中进行非刚体适应，通过NIF2D保留细粒度发音器（手/脸）和协同发音，（iii）在条件视频扩散之前使用Pose-MAE风格的补全。这种分解注入了几何和语言先验，产生形状和时空一致性。在多个大规模数据集（ASL 50K、How2Sign、CSL News）上，SignMimic在视频质量、身份相似性和帧连续性方面达到最先进水平，同时在生成视频上执行反向翻译（SLT）时实现最小损失。消融研究证实了刚体规范化、非刚体适应和补全的作用。代码、模型检查点和视频示例将发布。
英文摘要
We study the challenge of sign language video mimicking: given a driving video and a single reference frame, synthesize a video where the target signer reproduces the source motion while preserving identity and linguistic form. Prior pipelines entangle rigid motion, non-rigid deformation, and view-dependent completion in a monolithic generator, causing handshape drift and spatio-temporal instability. We present SignMimic, which (i) applies a TNet-based model to study SE(3) rigid canonicalization to stabilize global pose, (ii) performs non-rigid adaptation in a canonical space to preserve fine-grained articulators (hands/face) and coarticulation via NIF2D, and (iii) uses Pose-MAE-style completion before conditional video diffusion. This factorization injects geometric and linguistic priors, yielding shape and spatio-temporal consistency. On several large-scale datasets (ASL 50K, How2Sign, CSL News), SignMimic achieves state-of-the-art-level performance on video quality, identity similarity, and frame continuity while also achieving minimal loss when performing back translation (SLT) on generated videos. Ablations confirm the role of rigid canonicalization, non-rigid adaptation, and completion. Code is available at https://anonymous.4open.science/r/UniSignMimicTurbo-6088; model checkpoints and video examples will be released.
