arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2601.12876cs.CV
收藏

探索带有相邻帧先验的谈话头模型用于语音保留的面部表情操纵

Exploring Talking Head Models With Adjacent Frame Prior for Speech-Preserving Facial Expression Manipulation

Guangdong University of Technology（广东工业大学）；Peking University（北京大学）；Sun Yat-sen University（中山大学）

Zhenxuan Lu, Zhihua Xu, Zhijing Yang, Feng Gao, Yongyi Lu, Keze Wang, Tianshui Chen

备注：
Accepted by ACM Transactions on Multimedia Computing, Communications, and Applications
阅读 PDF
arXiv 页面 ↗HTML 原文 ↗

AI总结：

本文提出THFEM框架，结合AD-THG模型与SPFEM，通过相邻帧学习策略提升语音保留的面部表情操纵效果。

摘要：

语音保留的面部表情操纵（SPFEM）是一种创新技术，旨在在图像和视频中改变面部表情的同时保留原始嘴部运动。尽管有进展，SPFEM仍然在准确同步嘴唇运动方面存在困难，这归因于面部表情和嘴部形状之间复杂的相互作用。借助音频驱动的谈话头生成（AD-THG）模型在合成精确嘴唇运动方面的先进能力，我们的研究引入了这些模型与SPFEM的新整合。我们提出了一种新的框架，即谈话头面部表情操纵（THFEM），该框架利用AD-THG模型从音频输入生成具有准确同步嘴唇运动的帧，并利用SPFEM修改的图像。然而，增加AD-THG模型生成的帧数往往会损害图像的真实性和表情保真度。为此，我们开发了一种相邻帧学习策略，通过微调AD-THG模型来预测连续帧的序列。该策略使模型能够利用邻近帧的信息，显著提高测试期间的图像质量。我们的广泛实验评估表明，该框架在表情操纵过程中有效保持嘴部形状，突显了将AD-THG与SPFEM整合的显著优势。

英文摘要：

Speech-Preserving Facial Expression Manipulation (SPFEM) is an innovative technique aimed at altering facial expressions in images and videos while retaining the original mouth movements. Despite advancements, SPFEM still struggles with accurate lip synchronization due to the complex interplay between facial expressions and mouth shapes. Capitalizing on the advanced capabilities of audio-driven talking head generation (AD-THG) models in synthesizing precise lip movements, our research introduces a novel integration of these models with SPFEM. We present a new framework, Talking Head Facial Expression Manipulation (THFEM), which utilizes AD-THG models to generate frames with accurately synchronized lip movements from audio inputs and SPFEM-altered images. However, increasing the number of frames generated by AD-THG models tends to compromise the realism and expression fidelity of the images. To counter this, we develop an adjacent frame learning strategy that finetunes AD-THG models to predict sequences of consecutive frames. This strategy enables the models to incorporate information from neighboring frames, significantly improving image quality during testing. Our extensive experimental evaluations demonstrate that this framework effectively preserves mouth shapes during expression manipulations, highlighting the substantial benefits of integrating AD-THG with SPFEM.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。

↑