具有自适应音视频注意力的视觉感知音频字幕生成
Visually-Aware Audio Captioning With Adaptive Audio-Visual Attention
AI总结：
本文提出视觉感知音频字幕生成方法，通过引入视觉编码器和自适应音视频注意力机制，在AudioCaps数据集上取得了最先进的机器翻译指标结果。
摘要：
音频字幕生成旨在为音频片段生成文本描述。在现实世界中，许多物体会产生相似的声音。如何准确识别模糊声音是音频字幕生成面临的一大挑战。在本工作中，受人类固有跨模态感知的启发，我们提出视觉感知音频字幕生成，利用视觉信息来帮助描述发声模糊的物体。具体来说，我们引入一个现成的视觉编码器来提取视频特征，并将视觉特征融入音频字幕生成系统。此外，为了更好地利用互补的音视频上下文，我们提出一种音视频注意力机制，该机制自适应地整合音频和视觉上下文，并去除潜在空间中的冗余信息。在最大的音频字幕数据集AudioCaps上的实验结果表明，我们提出的方法在机器翻译指标上取得了最先进的结果。
英文摘要：
Audio captioning aims to generate text descriptions of audio clips. In the real world, many objects produce similar sounds. How to accurately recognize ambiguous sounds is a major challenge for audio captioning. In this work, inspired by inherent human multimodal perception, we propose visually-aware audio captioning, which makes use of visual information to help the description of ambiguous sounding objects. Specifically, we introduce an off-the-shelf visual encoder to extract video features and incorporate the visual features into an audio captioning system. Furthermore, to better exploit complementary audio-visual contexts, we propose an audio-visual attention mechanism that adaptively integrates audio and visual context and removes the redundant information in the latent space. Experimental results on AudioCaps, the largest audio captioning dataset, show that our proposed method achieves state-of-the-art results on machine translation metrics.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
