探索音频在视频字幕生成中的作用
Exploring the Role of Audio in Video Captioning
AI总结：
本文提出一种音视频字幕生成框架，利用原始音频信号而非ASR文本，通过模态平衡预训练损失和局部-全局融合机制，在四个数据集上显著提升性能，部分指标超越现有最优方法。
摘要：
近年来，视频字幕生成的研究重点在于设计能够同时处理视频和文本模态的架构，并利用带有文本转录的大规模视频数据集（如HowTo100M）进行预训练。尽管这些方法取得了显著改进，但音频模态在视频字幕生成中常被忽视。在本工作中，我们提出了一种音视频框架，旨在充分挖掘音频模态在字幕生成中的潜力。我们主张，与其依赖通过自动语音识别（ASR）提取的文本转录，不如直接使用原始音频信号进行学习，因为音频包含额外信息，如声学事件、说话人身份等。我们的贡献有两方面。首先，我们观察到，当同时使用视频和音频模态进行预训练时，模型会过度特化于音频模态，因为仅使用音频即可预测出真实标签（即文本转录）。我们提出了一种模态平衡预训练（MBP）损失函数来缓解这一问题，并显著提升了下游任务的性能。其次，我们细致研究了跨模态模块的不同设计选择，这些选择可能成为信息瓶颈并产生较差的结果。我们提出了新的局部-全局融合机制，以改善音频和视频之间的信息交换。我们在四个数据集上证明了利用音频模态带来的显著改进，甚至在某些指标上，在不依赖文本模态作为输入的情况下超越了现有最先进的方法。
英文摘要：
Recent focus in video captioning has been on designing architectures that can consume both video and text modalities, and using large-scale video datasets with text transcripts for pre-training, such as HowTo100M. Though these approaches have achieved significant improvement, the audio modality is often ignored in video captioning. In this work, we present an audio-visual framework, which aims to fully exploit the potential of the audio modality for captioning. Instead of relying on text transcripts extracted via automatic speech recognition (ASR), we argue that learning with raw audio signals can be more beneficial, as audio has additional information including acoustic events, speaker identity, etc. Our contributions are twofold. First, we observed that the model overspecializes to the audio modality when pre-training with both video and audio modality, since the ground truth (i.e., text transcripts) can be solely predicted using audio. We proposed a Modality Balanced Pre-training (MBP) loss to mitigate this issue and significantly improve the performance on downstream tasks. Second, we slice and dice different design choices of the cross-modal module, which may become an information bottleneck and generate inferior results. We proposed new local-global fusion mechanisms to improve information exchange across audio and video. We demonstrate significant improvements by leveraging the audio modality on four datasets, and even outperform the state of the art on some metrics without relying on the text modality as the input.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
