arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2108.05762cs.HCcs.LGcs.MM
收藏

手势特征可预测性的多模态分析

Multimodal analysis of the predictability of hand-gesture properties

KTH Royal Institute of Technology(瑞典皇家理工学院)

Taras Kucherenko, Rajmund Nagy, Michael Neff, Hedvig Kjellström, Gustav Eje Henter

摘要:

智能会话代理受益于能够用手势伴随他们的讲话。虽然近年来已经提出了许多数据驱动的手势生成方法,但仍然不清楚这些系统是否可以一致地生成传达含义的手势。我们研究了哪些手势属性(相位,类别和语义)可以使用当代深度学习从语音文本和/或音频中预测。在广泛的实验中,我们表明,手势的含义(语义和类别)相关的手势属性是可预测的文本功能(时间对齐的FastText嵌入)单独,但不是从韵律音频功能,而节奏相关的手势属性(相位)另一方面,可以预测音频功能比从文本。这些结果是令人鼓舞的,因为它们表明,它是可以配备一个具体的代理与内容明智的有意义的共同语音手势使用机器学习模型。

英文摘要:

Embodied conversational agents benefit from being able to accompany their speech with gestures. Although many data-driven approaches to gesture generation have been proposed in recent years, it is still unclear whether such systems can consistently generate gestures that convey meaning. We investigate which gesture properties (phase, category, and semantics) can be predicted from speech text and/or audio using contemporary deep learning. In extensive experiments, we show that gesture properties related to gesture meaning (semantics and category) are predictable from text features (time-aligned FastText embeddings) alone, but not from prosodic audio features, while rhythm-related gesture properties (phase) on the other hand can be predicted from audio features better than from text. These results are encouraging as they indicate that it is possible to equip an embodied agent with content-wise meaningful co-speech gestures using a machine-learning model.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑