META4:使用自监督文本和语音表征的语义对齐隐喻手势生成
META4: Semantically-Aligned Generation of Metaphoric Gestures Using Self-Supervised Text and Speech Representation
AI总结:
提出META4深度学习方法,通过从输入文本计算意象图式并结合语音,首次实现了语音驱动的语义对齐隐喻手势生成。
摘要:
意象图式是重复性的认知模式,影响着我们对语音中各种概念进行概念化和推理的方式。这些模式深深植根于我们的认知过程中,并反映在包括手势在内的身体表达上。特别是,隐喻手势具有与意象图式相一致的本质特征和语义含义,以视觉化地呈现抽象概念。手势的形状和形式能够传达抽象概念,例如伸展前臂和手,或通过手部动作描绘线条,以视觉化地呈现PATH(路径)的意象图式。以往的行为生成模型主要集中于利用语音(声学特征和文本)来驱动虚拟智能体的生成模型。它们未考虑意象图式所承载的关键语义信息,从而无法有效地生成隐喻手势。为了解决这一局限性,我们引入了META4,这是一种从语音和意象图式生成隐喻手势的深度学习方法。我们的方法有两个主要目标:从输入文本中计算意象图式以捕捉潜在的语义和隐喻含义,并在语音和计算出的意象图式的驱动下生成隐喻手势。我们的方法是首个利用意象图式潜力来生成语音驱动的隐喻手势的方法。我们证明了该方法的有效性,并强调了语音和意象图式在隐喻手势建模中的重要性。
英文摘要:
Image Schemas are repetitive cognitive patterns that influence the way we conceptualize and reason about various concepts present in speech. These patterns are deeply embedded within our cognitive processes and are reflected in our bodily expressions including gestures. Particularly, metaphoric gestures possess essential characteristics and semantic meanings that align with Image Schemas, to visually represent abstract concepts. The shape and form of gestures can convey abstract concepts, such as extending the forearm and hand or tracing a line with hand movements to visually represent the image schema of PATH. Previous behavior generation models have primarily focused on utilizing speech (acoustic features and text) to drive the generation model of virtual agents. They have not considered key semantic information as those carried by Image Schemas to effectively generate metaphoric gestures. To address this limitation, we introduce META4, a deep learning approach that generates metaphoric gestures from both speech and Image Schemas. Our approach has two primary goals: computing Image Schemas from input text to capture the underlying semantic and metaphorical meaning, and generating metaphoric gestures driven by speech and the computed image schemas. Our approach is the first method for generating speech driven metaphoric gestures while leveraging the potential of Image Schemas. We demonstrate the effectiveness of our approach and highlight the importance of both speech and image schemas in modeling metaphoric gestures.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。