用于空间声音重现的视频中视听说话人定位
Audio-Visual Talker Localization in Video for Spatial Sound Reproduction
AI总结：
本研究提出融合视觉与多通道音频的视听方法，用于视频中说话人定位与位置元数据提取，相比传统方法显著降低了检测误差并提升了空间精度。
摘要：
基于对象的音频制作需要为每个点声源对象定义位置元数据，包括声音场景前景中的关键元素。在许多媒体制作场景中，通常会同时使用摄像机和麦克风进行录制，而人声往往是关键元素。在本研究中，我们检测并定位视频中的活跃说话人，从而实现对说话人相对于摄像机参考系的位置元数据的自动提取。通过引入视觉模态，本研究在我们之前仅关注基于音频的活跃说话人检测与定位研究的基础上进行了扩展。我们的实验比较了利用单声道音频的传统视听活跃说话人检测方法、我们之前利用麦克风阵列多通道录音的纯音频方法，以及一种融合视觉与多通道音频的新型视听方法。我们发现这两种模态起到了互补作用。多通道音频克服了视觉遮挡问题，与使用单通道音频的视听方法相比，检测误差降低了两位数。多通道音频与视觉的结合进一步提升了空间精度，在 Tragic Talkers 数据集上 F1 分数提高了 4 个百分点。未来的研究将评估模型在嘈杂和高混响环境中的鲁棒性，并解决屏幕外说话人的问题。
英文摘要：
Object-based audio production requires the positional metadata to be defined for each point-source object, including the key elements in the foreground of the sound scene. In many media production use cases, both cameras and microphones are employed to make recordings, and the human voice is often a key element. In this research, we detect and locate the active speaker in the video, facilitating the automatic extraction of the positional metadata of the talker relative to the camera's reference frame. With the integration of the visual modality, this study expands upon our previous investigation focused solely on audio-based active speaker detection and localization. Our experiments compare conventional audio-visual approaches for active speaker detection that leverage monaural audio, our previous audio-only method that leverages multichannel recordings from a microphone array, and a novel audio-visual approach integrating vision and multichannel audio. We found the role of the two modalities to complement each other. Multichannel audio, overcoming the problem of visual occlusions, provides a double-digit reduction in detection error compared to audio-visual methods with single-channel audio. The combination of multichannel audio and vision further enhances spatial accuracy, leading to a four-percentage point increase in F1 score on the Tragic Talkers dataset. Future investigations will assess the robustness of the model in noisy and highly reverberant environments, as well as tackle the problem of off-screen speakers.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
