在大规模开放挑战赛中评估手势生成:GENEA Challenge 2022
Evaluating gesture generation in a large-scale open challenge: The GENEA Challenge 2022
AI总结:
GENEA Challenge 2022 通过统一数据集、渲染流程和众包主观评估,对十支团队的共语音手势生成系统进行基准比较,揭示合成动作可在拟人度上超过动捕但语音适当性仍明显不足。
摘要:
本文报告了第二届 GENEA Challenge,该挑战赛旨在对数据驱动的自动共语音手势生成进行基准测试。参赛团队使用相同的语音和动作数据集构建手势生成系统。所有这些系统生成的动作都通过标准化可视化流程渲染为视频,并在若干大规模众包用户研究中接受评估。与比较不同研究论文时不同,此处结果差异仅源于方法之间的差异,因此能够在各系统之间进行直接比较。该数据集基于18小时的全身动作捕捉(包含手指),记录了不同人员进行双人对话的过程。共有十支团队参加挑战赛,比赛分为两个层级:全身手势和上半身手势。对于每个层级,我们都评估了手势动作的拟人程度及其对特定语音信号的适当性。我们的评估将拟人程度与手势适当性解耦,而这一直是该领域的一个难题。\n\n评估结果表明,一些合成手势条件被评定为显著比3D人体动作捕捉更像人类。据我们所知,此前尚未有人证明这一点。另一方面,研究发现,与原始动作捕捉记录相比,所有合成动作对语音的适当性都大幅降低。我们还发现,在这项大规模评估中,传统客观指标与主观拟人程度评分的相关性并不好。唯一的例外是 Fréchet gesture distance(FGD),其 Kendall's tau 秩相关约为 -0.5。基于挑战赛结果,我们为系统构建和评估提出了大量建议。
英文摘要:
This paper reports on the second GENEA Challenge to benchmark data-driven automatic co-speech gesture generation. Participating teams used the same speech and motion dataset to build gesture-generation systems. Motion generated by all these systems was rendered to video using a standardised visualisation pipeline and evaluated in several large, crowdsourced user studies. Unlike when comparing different research papers, differences in results are here only due to differences between methods, enabling direct comparison between systems. The dataset was based on 18 hours of full-body motion capture, including fingers, of different persons engaging in a dyadic conversation. Ten teams participated in the challenge across two tiers: full-body and upper-body gesticulation. For each tier, we evaluated both the human-likeness of the gesture motion and its appropriateness for the specific speech signal. Our evaluations decouple human-likeness from gesture appropriateness, which has been a difficult problem in the field. The evaluation results show some synthetic gesture conditions being rated as significantly more human-like than 3D human motion capture. To the best of our knowledge, this has not been demonstrated before. On the other hand, all synthetic motion is found to be vastly less appropriate for the speech than the original motion-capture recordings. We also find that conventional objective metrics do not correlate well with subjective human-likeness ratings in this large evaluation. The one exception is the Fréchet gesture distance (FGD), which achieves a Kendall's tau rank correlation of around $-0.5$. Based on the challenge results we formulate numerous recommendations for system building and evaluation.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。