ReactMotion: 从说话者 utterance 生成反应性听众动作
ReactMotion: Generating Reactive Listener Motions from Speaker Utterance
AI总结：
本文提出反应性听众动作生成任务，通过ReactMotionNet数据集和偏好导向评估协议，开发统一生成框架，生成更自然、多样且恰当的听众动作。
摘要：
本文介绍了一个新任务，即从说话者 utterance 生成反应性听众动作，旨在生成自然的听众身体动作，以适当响应说话者的 utterance。然而，建模此类非语言听众行为仍处于探索和挑战阶段，由于人类反应本质上是非确定性的。为此，我们提出了ReactMotionNet，一个大规模数据集，将说话者 utterance 与多个候选听众动作配对，并标记不同程度的适当性。该数据集设计明确捕捉了听众行为的一对多性质，并提供了超越单一真实动作的监督。基于此数据集设计，我们开发了针对评估反应适当性的偏好导向评估协议，其中传统动作度量标准聚焦于输入-动作对齐而忽略。我们进一步提出了ReactMotion，一个统一的生成框架，联合建模文本、音频、情绪和动作，并通过基于偏好的目标进行训练，以鼓励适当且多样的听众反应。大量实验表明，ReactMotion在检索基线和级联LLM管道上表现更优，生成更自然、多样且恰当的听众动作。
英文摘要：
In this paper, we introduce a new task, Reactive Listener Motion Generation from Speaker Utterance, which aims to generate naturalistic listener body motions that appropriately respond to a speaker's utterance. However, modeling such nonverbal listener behaviors remains underexplored and challenging due to the inherently non-deterministic nature of human reactions. To facilitate this task, we present ReactMotionNet, a large-scale dataset that pairs speaker utterances with multiple candidate listener motions annotated with varying degrees of appropriateness. This dataset design explicitly captures the one-to-many nature of listener behavior and provides supervision beyond a single ground-truth motion. Building on this dataset design, we develop preference-oriented evaluation protocols tailored to evaluate reactive appropriateness, where conventional motion metrics focusing on input-motion alignment ignore. We further propose ReactMotion, a unified generative framework that jointly models text, audio, emotion, and motion, and is trained with preference-based objectives to encourage both appropriate and diverse listener responses. Extensive experiments show that ReactMotion outperforms retrieval baselines and cascaded LLM-based pipelines, generating more natural, diverse, and appropriate listener motions.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
