DBMIF:一种用于空气传导和骨传导语音增强的深度平衡多模态迭代融合框架
DBMIF: a deep balanced multimodal iterative fusion framework for air- and bone-conduction speech enhancement
AI总结:
DBMIF通过深度平衡多模态迭代融合框架提升空气传导和骨传导语音增强性能,降低字符错误率2.5%以上。
摘要:
传统语音增强系统的性能在极低信噪比(SNR)环境中急剧下降,尤其是在空气传导(AC)麦克风被环境噪声淹没的情况下。尽管骨传导(BC)传感器提供互补且噪声容忍的信息,但现有融合方法难以在广泛的SNR条件下保持一致的性能。为了解决这一限制,我们提出了深度平衡多模态迭代融合框架(DBMIF),一种三分支架构,旨在通过严格的跨模态交互重建高质量语音。具体而言,基于多尺度交互编码器解码器主干,该框架 orchestrate 一个迭代注意力模块和一个跨分支门控模块,以促进自适应加权和双向交换。为了补充这种动态交互,进一步集成了一个平衡交互瓶颈,以学习紧凑且稳定的融合表示。广泛的实验表明,DBMIF在不同噪声类型下在语音质量和可懂度方面均与最近的单模态和多模态基线具有竞争力。在下游ASR任务中,所提出的方法比竞争方法至少减少了2.5个百分点的字符错误率。这些结果证实,DBMIF有效地利用了BC语音的鲁棒性,同时保持AC语音的自然性,确保在现实世界场景中的可靠性。源代码可在github.com/wyl516w/dbmif上公开获取。
英文摘要:
The performance of conventional speech enhancement systems degrades sharply in extremely low signal-to-noise ratio (SNR) environments where air-conduction (AC) microphones are overwhelmed by ambient noise. Although bone-conduction (BC) sensors offer complementary, noise-tolerant information, existing fusion approaches struggle to maintain consistent performance across a wide range of SNR conditions. To address this limitation, we propose the Deep Balanced Multimodal Iterative Fusion Framework (DBMIF), a three-branch architecture designed to reconstruct high-fidelity speech through rigorous cross-modal interaction. Specifically, grounded in a multi-scale interactive encoder-decoder backbone, the framework orchestrates an iterative attention module and a cross-branch gated module to facilitate adaptive weighting and bidirectional exchange. To complement this dynamic interaction, a balanced-interaction bottleneck is further integrated to learn a compact, stable fused representation. Extensive experiments demonstrate that DBMIF achieves competitive performance compared with recent unimodal and multimodal baselines in both speech quality and intelligibility across diverse noise types. In downstream ASR tasks, the proposed method reduces the character error rate by at least 2.5 percent compared to competing approaches. These results confirm that DBMIF effectively harnesses the robustness of BC speech while preserving the naturalness of AC speech, ensuring reliability in real-world scenarios. The source code is publicly available at github.com/wyl516w/dbmif.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。