arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2405.07974cs.CV
收藏

SignAvatar:手语3D动作重建与生成

SignAvatar: Sign Language 3D Motion Reconstruction and Generation

University at Buffalo(布法罗大学)

Lu Dong, Lipisha Chaudhary, Fei Xu, Xiao Wang, Mason Lary, Ifeoma Nwogu

AI总结:

针对孤立手语单词3D动作重建与生成的难点,提出基于transformer条件变分自编码器的SignAvatar框架,结合课程学习提升效果,同时发布ASL3DWord数据集,实验验证其性能优越。

摘要:

由于缺乏真实世界的3D手语单词数据、手语动作存在复杂细微差别,且手语语义存在跨模态理解难题,实现孤立手语单词的表现力丰富的3D动作重建与自动生成颇具挑战。为应对这些难题,我们推出SignAvatar,一个兼具单词级手语重建与生成能力的框架。SignAvatar采用基于transformer的条件变分自编码器架构,有效建立不同语义模态间的关联。此外,该方法融入课程学习策略以提升模型的鲁棒性与泛化能力,生成更逼真的动作。我们还贡献了ASL3DWord数据集,包含对应不同独立手语单词的身体、手部及面部3D关节旋转数据。通过大量实验我们验证了SignAvatar的有效性,展现出其卓越的重建与自动生成能力。代码与数据集可在项目页面获取。

英文摘要:

Achieving expressive 3D motion reconstruction and automatic generation for isolated sign words can be challenging, due to the lack of real-world 3D sign-word data, the complex nuances of signing motions, and the cross-modal understanding of sign language semantics. To address these challenges, we introduce SignAvatar, a framework capable of both word-level sign language reconstruction and generation. SignAvatar employs a transformer-based conditional variational autoencoder architecture, effectively establishing relationships across different semantic modalities. Additionally, this approach incorporates a curriculum learning strategy to enhance the model's robustness and generalization, resulting in more realistic motions. Furthermore, we contribute the ASL3DWord dataset, composed of 3D joint rotation data for the body, hands, and face, for unique sign words. We demonstrate the effectiveness of SignAvatar through extensive experiments, showcasing its superior reconstruction and automatic generation capabilities. The code and dataset are available on the project page.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑