Tiny is not small enough: 高质量、低资源面部动画模型通过混合知识蒸馏
Tiny is not small enough: High-quality, low-resource facial animation models through hybrid knowledge distillation
AI总结：
本文通过混合知识蒸馏与伪标签方法，提出了一种设备端实时面部动画模型，显著减少内存占用并保持高质量动画，为模型驱动的数字角色发展提供新方向。
摘要：
高质量、鲁棒的机器学习模型训练用于语音驱动的3D面部动画需要一个大规模、多样化的高质量音频-动画配对数据集。为克服缺乏此类数据集的难题，近期工作引入了大型预训练语音编码器，这些编码器对输入音频的变化具有鲁棒性，因此使面部动画模型能够在不同说话人、音频质量和语言之间泛化。然而，由此产生的面部动画模型过于庞大，只能在专用机器上进行离线推断。在本工作中，我们探索了游戏开发中的设备端、实时面部动画模型。我们通过混合知识蒸馏与伪标签的方法克服缺乏大规模数据集的难题。给定一个大规模音频数据集，我们使用高性能的教师模型来训练非常小的学生模型。与预训练语音编码器不同，我们的学生模型仅包含卷积和全连接层，无需注意力上下文或递归更新。在我们的实验中，我们证明可以将内存占用减少到最高3.4 MB，并将所需的未来音频上下文减少到最高81毫秒，同时保持高质量的动画。这为设备端推断铺平了道路，是朝着逼真、模型驱动的数字角色的重要一步。
英文摘要：
The training of high-quality, robust machine learning models for speech-driven 3D facial animation requires a large, diverse dataset of high-quality audio-animation pairs. To overcome the lack of such a dataset, recent work has introduced large pre-trained speech encoders that are robust to variations in the input audio and, therefore, enable the facial animation model to generalize across speakers, audio quality, and languages. However, the resulting facial animation models are prohibitively large and lend themselves only to offline inference on a dedicated machine. In this work, we explore on-device, real-time facial animation models in the context of game development. We overcome the lack of large datasets by using hybrid knowledge distillation with pseudo-labeling. Given a large audio dataset, we employ a high-performing teacher model to train very small student models. In contrast to the pre-trained speech encoders, our student models only consist of convolutional and fully-connected layers, removing the need for attention context or recurrent updates. In our experiments, we demonstrate that we can reduce the memory footprint to up to 3.4 MB and required future audio context to up to 81 ms while maintaining high-quality animations. This paves the way for on-device inference, an important step towards realistic, model-driven digital characters.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
