JEP-KD：基于联合嵌入预测架构的知识蒸馏用于视觉语音识别
JEP-KD: Joint-Embedding Predictive Architecture Based Knowledge Distillation for Visual Speech Recognition
AI总结：
本文提出JEP-KD方法，通过在嵌入层引入生成网络并利用预训练ASR音频特征进行知识蒸馏，配合多模态多阶段训练，有效缩小视觉语音识别与自动语音识别的性能差距，并验证了其跨平台通用性。
摘要：
视觉语音识别（VSR）任务通常被认为比自动语音识别（ASR）具有更低的理论性能上限，这是由视觉传递语义信息的固有限制所致。为缓解这一挑战，本文引入了一种先进的、使用联合嵌入预测架构（JEPA）的知识蒸馏方法，命名为JEP-KD，旨在模型训练期间更有效地利用音频特征。JEP-KD的核心是在嵌入层中加入一个生成网络，该网络增强了视频编码器的语义特征提取能力，并使其与预训练ASR模型编码器的音频特征更加对齐。该方法旨在逐步缩小VSR与ASR之间的性能差距。此外，为JEP-KD框架建立了一套全面的多模态、多阶段训练方案，增强了训练过程的鲁棒性和有效性。实验结果表明，JEP-KD显著提高了VSR模型的性能，并在不同的VSR平台上展现出通用性，表明其在其他多模态任务中具有更广泛应用的潜力。
英文摘要：
Visual Speech Recognition (VSR) tasks are generally recognized to have a lower theoretical performance ceiling than Automatic Speech Recognition (ASR), owing to the inherent limitations of conveying semantic information visually. To mitigate this challenge, this paper introduces an advanced knowledge distillation approach using a Joint-Embedding Predictive Architecture (JEPA), named JEP-KD, designed to more effectively utilize audio features during model training. Central to JEP-KD is the inclusion of a generative network within the embedding layer, which enhances the video encoder's capacity for semantic feature extraction and brings it into closer alignment with the audio features from a pre-trained ASR model's encoder. This approach aims to progressively reduce the performance gap between VSR and ASR. Moreover, a comprehensive multimodal, multistage training regimen for the JEP-KD framework is established, bolstering the robustness and efficacy of the training process. Experiment results demonstrate that JEP-KD significantly improves the performance of VSR models and demonstrates versatility across different VSR platforms, indicating its potential for broader application within other multimodal tasks.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
