DiffSHEG：一种基于扩散的实时语音驱动整体3D表情与手势生成方法
DiffSHEG: A Diffusion-Based Approach for Real-Time Speech-driven Holistic 3D Expression and Gesture Generation
AI总结：
提出 DiffSHEG，用扩散 transformer 与外扩采样实现任意长度、实时同步的语音驱动3D表情和手势生成，并在两个公开数据集及用户研究中达到最优。
摘要：
我们提出 DiffSHEG，这是一种基于扩散的方法，用于由语音驱动、长度任意的整体3D表情与手势生成。以往工作分别关注协同语音手势或表情生成，而同步表情与手势的联合生成仍鲜有研究。为解决该问题，我们基于扩散的协同语音运动生成 transformer 实现了从表情到手势的单向信息流，从而促进表情—手势联合分布的更好匹配。此外，我们提出一种基于 outpainting（外扩）的采样策略，用于在扩散模型中生成任意长序列，兼具灵活性与计算效率。我们的方法提供了一种实用方案，可生成由语音驱动的高质量同步表情和手势。在两个公开数据集上的评估表明，本方法在定量和定性方面均达到最先进性能。此外，一项用户研究证实 DiffSHEG 优于以往方法。通过实现富有表现力且同步动作的实时生成，DiffSHEG 展现了其在数字人与具身智能体开发中各类应用的潜力。
英文摘要：
We propose DiffSHEG, a Diffusion-based approach for Speech-driven Holistic 3D Expression and Gesture generation with arbitrary length. While previous works focused on co-speech gesture or expression generation individually, the joint generation of synchronized expressions and gestures remains barely explored. To address this, our diffusion-based co-speech motion generation transformer enables uni-directional information flow from expression to gesture, facilitating improved matching of joint expression-gesture distributions. Furthermore, we introduce an outpainting-based sampling strategy for arbitrary long sequence generation in diffusion models, offering flexibility and computational efficiency. Our method provides a practical solution that produces high-quality synchronized expression and gesture generation driven by speech. Evaluated on two public datasets, our approach achieves state-of-the-art performance both quantitatively and qualitatively. Additionally, a user study confirms the superiority of DiffSHEG over prior approaches. By enabling the real-time generation of expressive and synchronized motions, DiffSHEG showcases its potential for various applications in the development of digital humans and embodied agents.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
