arXiv 2507.233432025-07-31cs.CVeess.IV

谁是更优的说话者：AI生成说话人脸的主客观质量评估

Who is a Better Talker: Subjective and Objective Quality Assessment for AI-Generated Talking Heads

Shanghai Jiao Tong University（上海交通大学）；PengCheng Laboratory（鹏城实验室）；Harvard Medical School（哈佛医学院）

Yingjie Zhou, Jiezhang Cao, Zicheng Zhang, Farong Wen, Yanwei Jiang, Jun Jia, Xiaohong Liu, Xiongkuo Min, Guangtao Zhai

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

针对AI生成说话人脸质量评估研究不足的问题，本文构建了规模最大的THQA-10K评估数据集，开展主观评分实验，并提出基于首帧等特征的客观评估方法，达到SOTA性能。

摘要：

驱动肖像的语音驱动方法因能够合成说话嘴型与面部动作，被形象地称为“Talkers（说话模型）”。尤其是随着Text-to-Image（T2I，文本生成图像）模型的快速发展，AI-Generated Talking Heads（AGTHs，AI生成说话人脸）已逐渐成为一种新兴的数字人媒体。然而，这些说话模型及其生成的AGTH的质量仍存在挑战，针对这些问题的综合研究仍然有限。为填补这一空白，本文提出了迄今为止规模最大的AGTH质量评估数据集THQA-10K，该数据集选取12个主流T2I模型和14个先进说话模型，针对14个提示词生成AGTH。在剔除AGTH生成失败的样本后，THQA-10K数据集包含10457个AGTH。随后，研究招募志愿者对AGTH进行主观评分并给出对应的失真类别。在对主观实验结果的分析中，我们从泛化性和质量两方面评估了说话模型的性能，同时揭示了现有AGTH存在的失真问题。最后，本文提出了一种基于首帧、Y-T切片以及色调-唇部一致性的客观质量评估方法。实验结果表明，该方法在AGTH质量评估任务上可达到state-of-the-art（SOTA，当前最优）性能。相关工作已发布于https://github.com/zyj-2000/Talker。

英文摘要：

Speech-driven methods for portraits are figuratively known as "Talkers" because of their capability to synthesize speaking mouth shapes and facial movements. Especially with the rapid development of the Text-to-Image (T2I) models, AI-Generated Talking Heads (AGTHs) have gradually become an emerging digital human media. However, challenges persist regarding the quality of these talkers and AGTHs they generate, and comprehensive studies addressing these issues remain limited. To address this gap, this paper presents the largest AGTH quality assessment dataset THQA-10K to date, which selects 12 prominent T2I models and 14 advanced talkers to generate AGTHs for 14 prompts. After excluding instances where AGTH generation is unsuccessful, the THQA-10K dataset contains 10,457 AGTHs. Then, volunteers are recruited to subjectively rate the AGTHs and give the corresponding distortion categories. In our analysis for subjective experimental results, we evaluate the performance of talkers in terms of generalizability and quality, and also expose the distortions of existing AGTHs. Finally, an objective quality assessment method based on the first frame, Y-T slice and tone-lip consistency is proposed. Experimental results show that this method can achieve state-of-the-art (SOTA) performance in AGTH quality assessment. The work is released at https://github.com/zyj-2000/Talker.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。