多通道AV-wav2vec2：一个学习多通道多模态语音表示的框架
Multichannel AV-wav2vec2: A Framework for Learning Multichannel Multi-Modal Speech Representation
AI总结：
针对远场多通道语音处理的数据稀缺与噪声问题，提出多通道多模态自监督框架AV-wav2vec2，通过多路径结构与对比学习融合音视频数据，在AVSR、ASR、VSR和AVSR任务上验证了有效性。
摘要：
近年来，自监督语音预训练方法发展迅速，对许多近场单通道语音任务非常有效。然而，远场多通道语音处理面临着标注多通道数据稀缺和复杂环境噪声的问题。自监督学习在远场多通道和多模态语音处理中的有效性尚未得到充分探索。考虑到视觉信息有助于提高噪声场景中的语音识别性能，在本工作中，我们提出了一种多通道多模态语音自监督学习框架AV-wav2vec2，该框架利用视频和多通道音频数据作为输入。首先，我们提出了一种多路径结构来并行处理多通道音频流和视觉流，并以通道内和通道间对比损失作为训练目标，以充分利用多通道语音数据中的时空信息。其次，基于对比学习，我们使用了额外的单通道音频数据进行联合训练，以提升语音表示的性能。最后，我们使用真实场景下的中文多通道多模态数据集，验证了所提方法在视听语音识别（AVSR）、自动语音识别（ASR）、视觉语音识别（VSR）和视听说话人日志（AVSD）任务上的有效性。
英文摘要：
Self-supervised speech pre-training methods have developed rapidly in recent years, which show to be very effective for many near-field single-channel speech tasks. However, far-field multichannel speech processing is suffering from the scarcity of labeled multichannel data and complex ambient noises. The efficacy of self-supervised learning for far-field multichannel and multi-modal speech processing has not been well explored. Considering that visual information helps to improve speech recognition performance in noisy scenes, in this work we propose a multichannel multi-modal speech self-supervised learning framework AV-wav2vec2, which utilizes video and multichannel audio data as inputs. First, we propose a multi-path structure to process multichannel audio streams and a visual stream in parallel, with intra- and inter-channel contrastive losses as training targets to fully exploit the spatiotemporal information in multichannel speech data. Second, based on contrastive learning, we use additional single-channel audio data, which is trained jointly to improve the performance of speech representation. Finally, we use a Chinese multichannel multi-modal dataset in real scenarios to validate the effectiveness of the proposed method on audio-visual speech recognition (AVSR), automatic speech recognition (ASR), visual speech recognition (VSR) and audio-visual speaker diarization (AVSD) tasks.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
