基于扩散反演的统一协同语音手势生成编辑方法
A Unified Editing Method for Co-Speech Gesture Generation via Diffusion Inversion
AI总结：
本文提出一种基于扩散反演的统一框架，无需重新训练即可实现协同语音手势生成的高层与低层编辑，并通过实验验证其有效性。
摘要：
扩散模型在以语音音频或文本为条件、从含噪输入中为交互式人形机器人或数字虚拟形象生成高质量协同语音手势方面取得了巨大成功。然而，除了风格条件化等高层专用手段外，这些模型很少为内容创作者提供丰富的编辑能力。为解决这一问题，我们提出了一种利用扩散反演的统一框架，无需重新训练即可实现协同语音手势生成的多层次编辑能力。该方法利用了可逆扩散模型的两个关键能力。第一，通过反演，我们可以从手势中重建中间噪声，并从该噪声重新生成新手势。这可用于在不同语音条件下获得与原始手势具有高层相似性的手势。第二，这种重建降低了梯度计算过程中的激活缓存需求，使得在当前内存有限的硬件上对输入噪声进行直接优化成为可能。通过为例如关节旋转或速度设计不同的损失函数，我们可以通过优化自动调整输入噪声来控制各种低层细节。在多个用例上的大量实验表明，该框架成功统一了高层和低层的协同语音手势编辑。
英文摘要：
Diffusion models have shown great success in generating high-quality co-speech gestures for interactive humanoid robots or digital avatars from noisy input with the speech audio or text as conditions. However, they rarely focus on providing rich editing capabilities for content creators other than high-level specialized measures like style conditioning. To resolve this, we propose a unified framework utilizing diffusion inversion that enables multi-level editing capabilities for co-speech gesture generation without re-training. The method takes advantage of two key capabilities of invertible diffusion models. The first is that through inversion, we can reconstruct the intermediate noise from gestures and regenerate new gestures from the noise. This can be used to obtain gestures with high-level similarities to the original gestures for different speech conditions. The second is that this reconstruction reduces activation caching requirements during gradient calculation, making the direct optimization on input noises possible on current hardware with limited memory. With different loss functions designed for, e.g., joint rotation or velocity, we can control various low-level details by automatically tweaking the input noises through optimization. Extensive experiments on multiple use cases show that this framework succeeds in unifying high-level and low-level co-speech gesture editing.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
