Wav2Sem：用于3D语音驱动面部动画的即插即用音频语义解耦
Wav2Sem: Plug-and-Play Audio Semantic Decoupling for 3D Speech-Driven Facial Animation
AI总结：
本文提出即插即用的语义解相关模块Wav2Sem，通过提取音频语义特征对编码进行解耦，有效缓解了3D语音驱动面部动画中相似音节导致的唇形平均效应，提升了动画精确度与自然度。
摘要：
在3D语音驱动面部动画生成中，现有方法通常采用预训练的自监督音频模型作为编码器。然而，由于语言中普遍存在发音相似但唇形不同的音节，这些近同音音节在自监督音频特征空间中往往表现出显著耦合，导致后续唇部运动生成中出现平均效应。为解决此问题，本文提出即插即用的语义解相关模块——Wav2Sem。该模块提取与整个音频序列对应的语义特征，利用新增的语义信息在特征空间内对音频编码进行解相关，从而获得更具表现力的音频特征。在多个语音驱动模型上的大量实验表明，Wav2Sem模块有效解耦了音频特征，显著缓解了相似音节在唇形生成中的平均效应，进而提升了面部动画的精确度与自然度。源代码获取自 https://github.com/wslh852/Wav2Sem.git。
英文摘要：
In 3D speech-driven facial animation generation, existing methods commonly employ pre-trained self-supervised audio models as encoders. However, due to the prevalence of phonetically similar syllables with distinct lip shapes in language, these near-homophone syllables tend to exhibit significant coupling in self-supervised audio feature spaces, leading to the averaging effect in subsequent lip motion generation. To address this issue, this paper proposes a plug-and-play semantic decorrelation module-Wav2Sem. This module extracts semantic features corresponding to the entire audio sequence, leveraging the added semantic information to decorrelate audio encodings within the feature space, thereby achieving more expressive audio features. Extensive experiments across multiple Speech-driven models indicate that the Wav2Sem module effectively decouples audio features, significantly alleviating the averaging effect of phonetically similar syllables in lip shape generation, thereby enhancing the precision and naturalness of facial animations. Our source code is available at https://github.com/wslh852/Wav2Sem.git.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
