ReNeLiB:面向社交互动智能体的实时神经倾听行为生成
ReNeLiB: Real-time Neural Listening Behavior Generation for Socially Interactive Agents
AI总结:
本文提出首个名为ReNeLiB的实时软件工具包,通过多模态特征提取、神经网络行为生成及可视化组件,将机器学习方法集成至社交互动智能体框架,并引入心理治疗预训练模型,实现了实时且自然的非语言倾听行为生成。
摘要:
对人类行为做出灵活且自然的非语言反应,对于主要使用手工规则的社交互动智能体(SIAs)来说仍然是一个挑战。虽然最近提出的基于机器学习的对话行为生成方法是解决这一挑战的有前途的途径,但它们尚未被应用于SIAs。其主要原因是缺乏将此类方法与SIA框架集成,并符合人机交互场景苛刻实时性要求的软件工具包。在我们的工作中,我们首次提出了这样一个工具包,它由三个主要组件组成:(1)实时特征提取,用于从用户处捕捉多模态社交线索;(2)基于最新最先进神经网络方法的行为生成;(3)生成行为的可视化,支持基于FLAME和基于Apple ARKit的互动智能体。我们全面评估了整个框架及其组件的实时性能。此外,我们还引入了从心理治疗对话中得出的预训练行为生成模型,用于特定领域的倾听行为。我们的软件工具包对于实时部署和评估SIAs的倾听行为至关重要,现已公开可用。包括代码、从治疗互动中提取的行为多模态特征等资源,均托管在 https://daksitha.github.io/ReNeLib。
英文摘要:
Flexible and natural nonverbal reactions to human behavior remain a challenge for socially interactive agents (SIAs) that are predominantly animated using hand-crafted rules. While recently proposed machine learning based approaches to conversational behavior generation are a promising way to address this challenge, they have not yet been employed in SIAs. The primary reason for this is the lack of a software toolkit integrating such approaches with SIA frameworks that conforms to the challenging real-time requirements of human-agent interaction scenarios. In our work, we for the first time present such a toolkit consisting of three main components: (1) real-time feature extraction capturing multi-modal social cues from the user; (2) behavior generation based on a recent state-of-the-art neural network approach; (3) visualization of the generated behavior supporting both FLAME-based and Apple ARKit-based interactive agents. We comprehensively evaluate the real-time performance of the whole framework and its components. In addition, we introduce pre-trained behavioral generation models derived from psychotherapy sessions for domain-specific listening behaviors. Our software toolkit, pivotal for deploying and assessing SIAs' listening behavior in real-time, is publicly available. Resources, including code, behavioural multi-modal features extracted from therapeutic interactions, are hosted at https://daksitha.github.io/ReNeLib
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。