AVLnet：从教学视频中学习视听语言表征
AVLnet: Learning Audio-Visual Language Representations from Instructional Videos
摘要：
用于从视频学习视觉基础语言的当前方法通常依赖于文本注释，诸如人类生成的字幕或机器生成的自动语音识别（ASR）转录。在这项工作中，我们介绍了音频视频语言网络（AVLnet），这是一种自监督网络，可以直接从原始视频输入中学习共享的视听嵌入空间。为了避免文本注释的需要，我们从随机分割的视频片段及其原始音频波形中学习视听表示。我们训练AVLnet上HowTo100 M，一个大型的语料库的公开可用的教学视频，并评估图像检索和视频检索任务，达到最先进的性能。我们对AVLnet的学习表示进行了分析，表明我们的模型利用语音和自然声音来学习视听概念。此外，我们提出了一个三模态模型，联合处理原始音频，视频和文本字幕从视频学习多模态语义嵌入空间有用的文本视频检索。我们的代码、数据和训练模型将在avlnet.csail.mit.edu上发布
英文摘要：
Current methods for learning visually grounded language from videos often rely on text annotation, such as human generated captions or machine generated automatic speech recognition (ASR) transcripts. In this work, we introduce the Audio-Video Language Network (AVLnet), a self-supervised network that learns a shared audio-visual embedding space directly from raw video inputs. To circumvent the need for text annotation, we learn audio-visual representations from randomly segmented video clips and their raw audio waveforms. We train AVLnet on HowTo100M, a large corpus of publicly available instructional videos, and evaluate on image retrieval and video retrieval tasks, achieving state-of-the-art performance. We perform analysis of AVLnet's learned representations, showing our model utilizes speech and natural sounds to learn audio-visual concepts. Further, we propose a tri-modal model that jointly processes raw audio, video, and text captions from videos to learn a multi-modal semantic embedding space useful for text-video retrieval. Our code, data, and trained models will be released at avlnet.csail.mit.edu
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
