模态无关教师模型遇上弱监督视听事件解析器
Modality-Independent Teachers Meet Weakly-Supervised Audio-Visual Event Parser
AI总结：
针对弱监督非对齐视听事件解析任务，提出VALOR方法，利用模态无关对比预训练教师模型生成模态标签，在LLP数据集及视听事件定位任务上均刷新最优性能。
摘要：
视听学习一直是多模态机器学习的重要支柱，该领域研究大多聚焦于模态对齐场景，即假设音频和视觉模态均能传递预测目标的信号。我们借助Look, Listen, and Parse数据集（LLP，即“看、听与解析”数据集），探索研究尚不充分的非对齐场景：该场景的目标是仅通过弱标签识别视频中的音频和视觉事件。这类弱视频级标签仅能告知发生了哪些事件，却无法说明事件是通过哪种模态（音频、视觉，或两者兼具）被感知的。为提升这一极具挑战性场景下的学习效果，我们引入大规模对比预训练模型作为模态教师模型，并提出了一种简单、有效且通用的方法——Visual-Audio Label Elaboration（VALOR，即视听标签细化方法），用于为训练事件生成模态标签。实验研究表明，生成的标签使注意力基线模型的平均F值（Type@AV）显著提升8.0。令人意外的是，我们发现模态无关教师模型的表现优于模态融合教师模型，因为前者不会受到其他可能未对齐模态的噪声干扰。此外，我们的最优模型在LLP数据集的所有指标上均大幅刷新了现有最优水平（Type@AV的F值提升5.4）。VALOR还被进一步推广至视听事件定位任务，同样取得了新的最优结果。代码可在https://github.com/Franklin905/VALOR获取。
英文摘要：
Audio-visual learning has been a major pillar of multi-modal machine learning, where the community mostly focused on its modality-aligned setting, i.e., the audio and visual modality are both assumed to signal the prediction target. With the Look, Listen, and Parse dataset (LLP), we investigate the under-explored unaligned setting, where the goal is to recognize audio and visual events in a video with only weak labels observed. Such weak video-level labels only tell what events happen without knowing the modality they are perceived (audio, visual, or both). To enhance learning in this challenging setting, we incorporate large-scale contrastively pre-trained models as the modality teachers. A simple, effective, and generic method, termed Visual-Audio Label Elaboration (VALOR), is innovated to harvest modality labels for the training events. Empirical studies show that the harvested labels significantly improve an attentional baseline by 8.0 in average F-score (Type@AV). Surprisingly, we found that modality-independent teachers outperform their modality-fused counterparts since they are noise-proof from the other potentially unaligned modality. Moreover, our best model achieves the new state-of-the-art on all metrics of LLP by a substantial margin (+5.4 F-score for Type@AV). VALOR is further generalized to Audio-Visual Event Localization and achieves the new state-of-the-art as well. Code is available at: https://github.com/Franklin905/VALOR.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
