AD-AVSR：用于鲁棒视听语音识别的非对称双流增强
AD-AVSR: Asymmetric Dual-stream Enhancement for Robust Audio-Visual Speech Recognition
AI总结：
提出基于双向模态增强的视听语音识别框架AD-AVSR，通过音频双流编码、音频感知视觉精炼和跨模态噪声抑制掩蔽模块实现闭环信息流，并采用阈值机制过滤弱相关模态对，在LRS2和LRS3上超越SOTA。
摘要：
视听语音识别（AVSR）结合视听模态以改善语音识别，尤其在嘈杂环境中。然而，现有方法多采用单向增强或对称融合方式，限制了其捕获视听数据异构与互补相关性的能力，尤其在非对称信息条件下。为解决此问题，我们提出基于双向模态增强的AVSR框架AD-AVSR。具体而言，我们首先引入音频双流编码策略，从多视角丰富音频表征，并有意建立非对称性以支持后续跨模态交互。增强过程包含两个关键组件：音频感知视觉精炼模块在音频引导下增强视觉表征；跨模态噪声抑制掩蔽模块利用视觉线索精炼音频表征，二者协同形成闭环双向信息流。为进一步增强相关性鲁棒性，我们采用基于阈值的选择机制过滤无关或弱相关的视听对。在LRS2和LRS3数据集上的大量实验表明，AD-AVSR在性能和噪声鲁棒性上均持续超越SOTA方法，凸显了模型设计的有效性。
英文摘要：
Audio-visual speech recognition (AVSR) combines audio-visual modalities to improve speech recognition, especially in noisy environments. However, most existing methods deploy the unidirectional enhancement or symmetric fusion manner, which limits their capability to capture heterogeneous and complementary correlations of audio-visual data-especially under asymmetric information conditions. To tackle these gaps, we introduce a new AVSR framework termed AD-AVSR based on bidirectional modality enhancement. Specifically, we first introduce the audio dual-stream encoding strategy to enrich audio representations from multiple perspectives and intentionally establish asymmetry to support subsequent cross-modal interactions. The enhancement process involves two key components, Audio-aware Visual Refinement Module for enhanced visual representations under audio guidance, and Cross-modal Noise Suppression Masking Module which refines audio representations using visual cues, collaboratively leading to the closed-loop and bidirectional information flow. To further enhance correlation robustness, we adopt a threshold-based selection mechanism to filter out irrelevant or weakly correlated audio-visual pairs. Extensive experimental results on the LRS2 and LRS3 datasets indicate that our AD-AVSR consistently surpasses SOTA methods in both performance and noise robustness, highlighting the effectiveness of our model design.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
