面向整体同步言语手势生成的感知基础与语义感知评估

中文摘要 AI 辅助

整体且语义感知的同步言语手势生成已取得快速进展，但评估工作仍滞后：客观指标无法始终反映人类感知，且语义适配性难以量化。本文提出一种感知基础且语义感知的基准，结合标准化模型对比、以人为中心的指标验证及细粒度语义评估。首先整理涵盖分布相似性、几何保真度、运动质量、跨模态同步性、语义适配性等不同方面的13项客观指标；针对语义适配性类别，提出新指标语义手势保留度（Semantic Gesture Preservation, SGP），用于衡量生成手势对真实手势中语义手势的保留程度，为此利用多模态大语言模型（LLM）扩充BEAT2数据集的标注。随后开展感知研究，101名参与者从类人性、运动多样性、无动画错误、语音时序匹配、内容匹配五个维度对生成手势评分，系统分析客观指标与主观评分的相关性。与语义得分（Semantic Score, SC）无显著关联不同，SGP与语音感知的人类判断具有选择性对齐；构建与主观维度对齐的5项目标特定复合指标，这些复合指标在所有五个维度上提升了感知对齐度，在无动画错误和内容匹配维度提升最大，表明互补的客观信号比单一指标更能近似人类判断。总体而言，研究结果显示客观指标需经主观评估验证。

英文摘要

Holistic and semantics-aware co-speech gesture generation has advanced rapidly, yet evaluation remains behind: objective metrics do not consistently reflect human perception, and semantic appropriateness remains difficult to quantify. We present a perceptually grounded and semantics-aware benchmark that combines standardized model comparison, human-centered metric validation, and fine-grained semantic evaluation. We first curate a list of 13 objective metrics covering different aspects, including distributional similarity, geometric fidelity, kinematic quality, cross-modal synchrony, and semantic appropriateness. For the semantic-appropriateness category, we propose a new metric, Semantic Gesture Preservation (SGP), which measures how far semantic gestures in the ground truth are preserved in the generated gestures. For this, we augment the BEAT2 dataset's annotations using a multi-modal LLM. We then conduct a perceptual study where 101 participants score generated gestures among five dimensions, including human-likeness, motion diversity, absence of animation errors, speech timing and content match. We systematically analyze objective metric--subjective score correlations. Unlike Semantic Score (SC), which shows no significant association with the evaluated perceptual dimensions, SGP is selectively aligned with speech-aware human judgments. We construct five target-specific composite metrics aligned with the subjective dimensions. These composites improve perceptual alignment across all five dimensions, with the largest gains for absence of animation errors and content match, indicating that complementary objective signals can better approximate human judgments than individual metrics alone. Overall, our results show that objective metrics require validation against subjective evaluations.

发表机构

• Max Planck Institute for Psycholinguistics（马克斯·普朗克心理语言学研究所）
• Utrecht University（乌得勒支大学）
• Donders Institute for Brain Cognition and Behaviour（唐德斯脑认知与行为研究所）

机构由 AI 辅助整理，请以论文原文为准。