端到端视听语音活动检测的双模态循环神经网络模型
End-to-end Audiovisual Speech Activity Detection with Bimodal Recurrent Neural Models
AI总结：
本研究提出一种端到端双模态循环神经网络（BRNN）用于视听语音活动检测，直接从原始数据学习声学和视觉特征，在大型语料库上相比音频基线提升1.2%，并在噪声环境下达到92.7%的F1分数。
摘要：
语音活动检测（SAD）在当前语音处理系统中扮演重要角色，包括自动语音识别（ASR）。在存在声学噪声的环境中，SAD尤为困难。一个实用的解决方案是融合视觉信息，以增强SAD方法的鲁棒性。视听系统具有对不同语音模式（如耳语）或背景噪声鲁棒的优势。近年来，利用深度学习的视听语音处理进展为以原则性方式捕捉声学与视觉特征之间的时间关系开辟了机遇。本研究探索了这一思想，提出了一种用于SAD的\ufffd\ufffd双模态循环神经网络\ufffd\ufffd（BRNN）框架。该方法对顺序视听数据的时态动态进行建模，提高了所提出的SAD系统的准确性和鲁棒性。研究未采用手工设计的特征，而是探索了端到端训练方法，在训练期间直接从原始数据中学习声学和视觉特征。实验评估使用了一个大型视听语料库，包含超过60.8小时的录音，来自105位说话人。结果表明，所提出的框架在实际场景中相较于仅使用音频且基于深度神经网络（DNN）实现的VAD基线，取得了高达1.2%的绝对改进。所提出的方法在嘈杂声学环境下使用便携式平板电脑的传感器进行评估时，达到了92.7%的F1分数，仅比在理想条件下（例如，使用高清摄像头和近讲麦克风获得的清晰语音）的性能低1.0%。
英文摘要：
Speech activity detection (SAD) plays an important role in current speech processing systems, including automatic speech recognition (ASR). SAD is particularly difficult in environments with acoustic noise. A practical solution is to incorporate visual information, increasing the robustness of the SAD approach. An audiovisual system has the advantage of being robust to different speech modes (e.g., whisper speech) or background noise. Recent advances in audiovisual speech processing using deep learning have opened opportunities to capture in a principled way the temporal relationships between acoustic and visual features. This study explores this idea proposing a \emph{bimodal recurrent neural network} (BRNN) framework for SAD. The approach models the temporal dynamic of the sequential audiovisual data, improving the accuracy and robustness of the proposed SAD system. Instead of estimating hand-crafted features, the study investigates an end-to-end training approach, where acoustic and visual features are directly learned from the raw data during training. The experimental evaluation considers a large audiovisual corpus with over 60.8 hours of recordings, collected from 105 speakers. The results demonstrate that the proposed framework leads to absolute improvements up to 1.2% under practical scenarios over a VAD baseline using only audio implemented with deep neural network (DNN). The proposed approach achieves 92.7% F1-score when it is evaluated using the sensors from a portable tablet under noisy acoustic environment, which is only 1.0% lower than the performance obtained under ideal conditions (e.g., clean speech obtained with a high definition camera and a close-talking microphone).
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
