arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2202.05383cs.CLcs.CVcs.LG
收藏

将面部表情纳入手语生成的上下文嵌入

Including Facial Expressions in Contextual Embeddings for Sign Language Generation

Stella AI(斯特拉人工智能公司);Carnegie Mellon University(卡内基梅隆大学);University of Pittsburgh(匹兹堡大学);Gallaudet University(加劳德特大学)

Carla Viegas, Mert İnan, Lorna Quandt, Malihe Alikhani

AI总结:

本文提出双编码器Transformer,首次结合面部动作单元建模文本、注释与面部表情关系,以同时生成手部动作和面部表情,从而提升手语生成的表现力和自然度。

摘要:

当前最先进的手语生成框架缺乏表现力和自然度,原因在于它们只关注手部动作,而忽视了面部表情的情感、语法和语义功能。本研究的目的是通过对面部表情进行语义锚定来增强手语的语义表示。我们研究了建模文本、注释和面部表情之间关系对手语生成系统性能的影响。具体而言,我们提出了一种双编码器Transformer,通过捕捉文本和手语注释中发现的相似性与差异性,能够同时生成手部动作和面部表情。我们考虑了面部肌肉活动在表达手部动作强度中的作用,首次将面部动作单元用于手语生成。我们进行了一系列实验,结果表明所提出的模型提高了自动生成手语的质量。

英文摘要:

State-of-the-art sign language generation frameworks lack expressivity and naturalness which is the result of only focusing manual signs, neglecting the affective, grammatical and semantic functions of facial expressions. The purpose of this work is to augment semantic representation of sign language through grounding facial expressions. We study the effect of modeling the relationship between text, gloss, and facial expressions on the performance of the sign generation systems. In particular, we propose a Dual Encoder Transformer able to generate manual signs as well as facial expressions by capturing the similarities and differences found in text and sign gloss annotation. We take into consideration the role of facial muscle activity to express intensities of manual signs by being the first to employ facial action units in sign language generation. We perform a series of experiments showing that our proposed model improves the quality of automatically generated sign language.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑