Speech2AffectiveGestures:利用生成对抗式情感表达学习合成协同语音手势
Speech2AffectiveGestures: Synthesizing Co-Speech Gestures with Generative Adversarial Affective Expression Learning
AI总结:
本文提出Speech2AffectiveGestures,通过生成对抗网络与多尺度时空图卷积情感编码器,从语音和种子姿态合成具有适当情感表达的协同语音手势,并在多个基准上显著优于现有方法。
摘要:
我们提出了一种生成对抗网络,用于合成具有适当情感表达的协同语音上半身手势的三维姿态序列。我们的网络由两个组件组成:一个生成器,用于从输入语音和种子姿态编码的特征联合嵌入空间中合成手势;以及一个判别器,用于区分合成姿态序列与真实三维姿态序列。我们在生成器中利用从输入语音计算得到的梅尔频率倒谱系数和文本转录,分别在独立的编码器中学习所需的情感以及相关的情感线索。我们设计了一个情感编码器,使用多尺度时空图卷积将三维姿态序列转换为潜在的、基于姿态的情感特征。我们在生成器和判别器中都使用了该情感编码器:在生成器中,它从种子姿态中学习情感特征以指导手势合成;在判别器中,它强制合成的手势包含适当的情感表达。我们在两个用于从语音合成手势的基准数据集上进行了广泛评估,即TED Gesture Dataset和GENEA Challenge 2020 Dataset。与最佳基线相比,我们将平均关节绝对误差降低了10%至33%,平均加速度差异降低了8%至58%,Fréchet手势距离降低了21%至34%。我们还进行了一项用户研究,观察到与当前最佳基线相比,约15.28%的参与者认为我们合成的手势看起来更合理,约16.32%的参与者认为这些手势具有与语音更一致的适当情感表达。
英文摘要:
We present a generative adversarial network to synthesize 3D pose sequences of co-speech upper-body gestures with appropriate affective expressions. Our network consists of two components: a generator to synthesize gestures from a joint embedding space of features encoded from the input speech and the seed poses, and a discriminator to distinguish between the synthesized pose sequences and real 3D pose sequences. We leverage the Mel-frequency cepstral coefficients and the text transcript computed from the input speech in separate encoders in our generator to learn the desired sentiments and the associated affective cues. We design an affective encoder using multi-scale spatial-temporal graph convolutions to transform 3D pose sequences into latent, pose-based affective features. We use our affective encoder in both our generator, where it learns affective features from the seed poses to guide the gesture synthesis, and our discriminator, where it enforces the synthesized gestures to contain the appropriate affective expressions. We perform extensive evaluations on two benchmark datasets for gesture synthesis from the speech, the TED Gesture Dataset and the GENEA Challenge 2020 Dataset. Compared to the best baselines, we improve the mean absolute joint error by 10--33%, the mean acceleration difference by 8--58%, and the Fréchet Gesture Distance by 21--34%. We also conduct a user study and observe that compared to the best current baselines, around 15.28% of participants indicated our synthesized gestures appear more plausible, and around 16.32% of participants felt the gestures had more appropriate affective expressions aligned with the speech.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。