面向高效语音驱动人脸生成的统一压缩框架
A Unified Compression Framework for Efficient Speech-Driven Talking-Face Generation
AI总结：
针对语音驱动人脸生成模型计算负担高的问题，提出统一压缩框架，通过构建紧凑生成器、知识蒸馏训练及选择性量化，实现大幅参数缩减与边缘GPU加速，同时保留生成质量。
摘要：
虚拟人已在娱乐、电商等众多行业受到广泛关注。作为核心技术，利用生成对抗网络从目标语音和人脸身份合成具有照片级真实感的人脸帧的研究已得到积极开展。尽管现有的人脸生成模型已取得显著成果，但它们通常带来高计算负担，限制了其高效部署。本研究旨在开发一种用于语音驱动人脸合成的轻量级模型。我们通过移除残差块并缩减通道宽度，基于流行的人脸生成器Wav2Lip构建了一个紧凑生成器。我们还提出了一种知识蒸馏方案，无需对抗学习即可稳定且有效地训练小容量生成器。我们将参数量和乘加运算量（MACs）减少了28倍，同时保留了原始模型的性能。此外，为缓解将整个生成器转换为INT8精度时出现的严重性能下降，我们采用了一种选择性量化方法，对量化敏感层使用FP16精度，对其余层使用INT8精度。借助这种混合精度，我们在边缘GPU上实现了最高19倍的加速，且不会明显降低生成质量。
英文摘要：
Virtual humans have gained considerable attention in numerous industries, e.g., entertainment and e-commerce. As a core technology, synthesizing photorealistic face frames from target speech and facial identity has been actively studied with generative adversarial networks. Despite remarkable results of modern talking-face generation models, they often entail high computational burdens, which limit their efficient deployment. This study aims to develop a lightweight model for speech-driven talking-face synthesis. We build a compact generator by removing the residual blocks and reducing the channel width from Wav2Lip, a popular talking-face generator. We also present a knowledge distillation scheme to stably yet effectively train the small-capacity generator without adversarial learning. We reduce the number of parameters and MACs by 28$\times$ while retaining the performance of the original model. Moreover, to alleviate a severe performance drop when converting the whole generator to INT8 precision, we adopt a selective quantization method that uses FP16 for the quantization-sensitive layers and INT8 for the other layers. Using this mixed precision, we achieve up to a 19$\times$ speedup on edge GPUs without noticeably compromising the generation quality.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
