以眼代耳：利用音视频分布对齐借助图像字幕生成器实现零样本音频描述
An Eye for an Ear: Zero-shot Audio Description Leveraging an Image Captioner using Audiovisual Distribution Alignment
AI总结：
针对音视频模态鸿沟阻碍音频字幕生成的问题，提出通过对齐音频与图像token分布的方法，在不改动原有图像字幕组件的情况下实现无监督零样本音频描述，性能显著优于现有方法。
摘要：
多模态大语言模型推动了图像字幕生成领域的发展，这些模型在海量图像数据集上微调后，展现出对语义概念的深度理解。本研究表明，这种能力可被重新用于音频字幕生成任务：我们可利用图像-语言联合解码器，对包含音视频内容的视频中与图像序列关联的听觉内容进行描述，这一目标可通过多模态对齐实现。但该多模态对齐任务颇具难度，因为真实视频中的听觉元素与视觉元素存在固有差异。此外，多模态表示学习通常依赖对比学习，面临所谓“模态鸿沟”的挑战，阻碍了不同模态间的顺畅融合。本文提出一种全新方法，通过匹配音频主干网络生成的token分布与图像字幕生成器的token分布，来弥合音视频模态鸿沟。我们的方法将音频token分布与图像token分布对齐，使模型能以无监督方式完成零样本音频字幕生成，同时保留初始图像字幕生成组件不变。通过将图像编码器与对齐后的音频编码器结合或用后者替换前者，这种对齐方式支持仅音频或音视频混合输入。与现有方法相比，我们的方法在零样本音频字幕生成任务上取得了显著提升的性能。
英文摘要：
Multimodal large language models have fueled progress in image captioning. These models, fine-tuned on vast image datasets, exhibit a deep understanding of semantic concepts. In this work, we show that this ability can be re-purposed for audio captioning, where the joint image-language decoder can be leveraged to describe auditory content associated with image sequences within videos featuring audiovisual content. This can be achieved via multimodal alignment. Yet, this multimodal alignment task is non-trivial due to the inherent disparity between audible and visible elements in real-world videos. Moreover, multimodal representation learning often relies on contrastive learning, facing the challenge of the so-called modality gap which hinders smooth integration between modalities. In this work, we introduce a novel methodology for bridging the audiovisual modality gap by matching the distributions of tokens produced by an audio backbone and those of an image captioner. Our approach aligns the audio token distribution with that of the image tokens, enabling the model to perform zero-shot audio captioning in an unsupervised fashion while keeping the initial image captioning component unaltered. This alignment allows for the use of either audio or audiovisual input by combining or substituting the image encoder with the aligned audio encoder. Our method achieves significantly improved performances in zero-shot audio captioning, compared to existing approaches.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
