利用视觉监督实现基于阵列的主动说话人检测与定位
Leveraging Visual Supervision for Array-based Active Speaker Detection and Localization
AI总结：
本文提出利用视觉监督的自监督学生-教师训练方法，使多通道音频CRNN独立于视觉进行主动说话人检测与定位，在遮挡场景下超越视听方法并达到与传统监督训练相当的性能。
摘要：
传统的主动说话人检测（ASD）的视听方法通常依赖于预先提取的视觉人脸轨迹和相应的单通道音频来在视频中寻找说话人。因此，每当说话人的面部不可见时，这些方法往往会失效。我们证明，一个简单的音频卷积循环神经网络（CRNN），使用从多通道音频中提取的空间输入特征进行训练，可以独立于视觉模态执行水平方向的主动说话人检测与定位（ASDL）。为了解决生成此类系统训练所需的真实标注数据的时间和成本问题，我们提出了一种新的自监督训练流程，该流程采用了“学生-教师”学习方式。一个传统的预训练主动说话人检测器被用作“教师”网络，以提供说话人位置作为伪标签。多通道音频“学生”网络被训练以生成相同的结果。在推理时，学生网络能够泛化并定位到教师网络无法在视觉上检测到的被遮挡的说话人，从而在召回率上带来显著提升。在TragicTalkers数据集上的实验表明，使用所提出的自监督学习方法训练的音频网络可以超越典型的视听方法，并产生与昂贵的传统监督训练相竞争的结果。我们证明，在学习流程中引入最小的人工监督可以实现改进。通过更大的训练集以及将视觉与多通道音频系统集成，可能获得进一步的提升。
英文摘要：
Conventional audio-visual approaches for active speaker detection (ASD) typically rely on visually pre-extracted face tracks and the corresponding single-channel audio to find the speaker in a video. Therefore, they tend to fail every time the face of the speaker is not visible. We demonstrate that a simple audio convolutional recurrent neural network (CRNN) trained with spatial input features extracted from multichannel audio can perform simultaneous horizontal active speaker detection and localization (ASDL), independently of the visual modality. To address the time and cost of generating ground truth labels to train such a system, we propose a new self-supervised training pipeline that embraces a ``student-teacher'' learning approach. A conventional pre-trained active speaker detector is adopted as a ``teacher'' network to provide the position of the speakers as pseudo-labels. The multichannel audio ``student'' network is trained to generate the same results. At inference, the student network can generalize and locate also the occluded speakers that the teacher network is not able to detect visually, yielding considerable improvements in recall rate. Experiments on the TragicTalkers dataset show that an audio network trained with the proposed self-supervised learning approach can exceed the performance of the typical audio-visual methods and produce results competitive with the costly conventional supervised training. We demonstrate that improvements can be achieved when minimal manual supervision is introduced in the learning pipeline. Further gains may be sought with larger training sets and integrating vision with the multichannel audio system.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
