DCIM-AVSR：通过 Dual Conformer Interaction Module 实现高效音视语音识别
DCIM-AVSR : Efficient Audio-Visual Speech Recognition via Dual Conformer Interaction Module
AI总结：
针对传统 AVSR 模型参数多、训练与部署成本高的问题，本文提出 DCIM 双 Conformer 交互架构及选择性参数更新的预训练方法，在减少参数的同时提升识别效率与性能。
摘要：
语音识别是使机器能够解释和处理人类语音、将口语转换为文本或命令的技术。该技术对于虚拟助手、转录服务和通信工具等应用至关重要。Audio-Visual Speech Recognition（AVSR，音视语音识别）模型通过纳入唇部运动和面部表情等视觉模态，增强了传统语音识别，尤其是在嘈杂环境中的表现。尽管在大规模数据集上训练、具有大量参数的传统 AVSR 模型能够取得显著的准确率，甚至常常超越人类表现，但它们也伴随着高昂的训练成本和部署挑战。为解决这些问题，我们提出一种高效的 AVSR 模型，通过集成 Dual Conformer Interaction Module（DCIM，双 Conformer 交互模块）来减少参数数量。此外，我们提出一种预训练方法，通过选择性地更新参数进一步优化模型性能，从而显著提升效率。不同于要求系统独立学习音频与视觉模态之间层级关系的传统模型，我们的方法将这种区分直接纳入模型架构。该设计同时提升了效率和性能，为 AVSR 任务带来了更实用、更有效的解决方案。
英文摘要：
Speech recognition is the technology that enables machines to interpret and process human speech, converting spoken language into text or commands. This technology is essential for applications such as virtual assistants, transcription services, and communication tools. The Audio-Visual Speech Recognition (AVSR) model enhances traditional speech recognition, particularly in noisy environments, by incorporating visual modalities like lip movements and facial expressions. While traditional AVSR models trained on large-scale datasets with numerous parameters can achieve remarkable accuracy, often surpassing human performance, they also come with high training costs and deployment challenges. To address these issues, we introduce an efficient AVSR model that reduces the number of parameters through the integration of a Dual Conformer Interaction Module (DCIM). In addition, we propose a pre-training method that further optimizes model performance by selectively updating parameters, leading to significant improvements in efficiency. Unlike conventional models that require the system to independently learn the hierarchical relationship between audio and visual modalities, our approach incorporates this distinction directly into the model architecture. This design enhances both efficiency and performance, resulting in a more practical and effective solution for AVSR tasks.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
