STELLA：具有时空局部对齐的持续音频-视频预训练
STELLA: Continual Audio-Video Pre-training with Spatio-Temporal Localized Alignment
AI总结：
提出STELLA持续音频-视频预训练方法，通过局部补丁重要性评分和重放引导相关性评估，实现有效学习，在零样本检索中相对提升3.69%，内存减少约45%。
摘要：
在不断发展的世界中，持续学习多种音频-视频语义对于音频相关的推理任务至关重要。然而，这是一个非平凡的问题，并带来了两个关键挑战：音频-视频对之间的稀疏时空相关性，以及会遗忘音频-视频关系的多模态相关性覆盖。为了解决这个问题，我们提出了一种新的持续音频-视频预训练方法，包含两个新颖的想法：（1）局部补丁重要性评分：我们引入一个多模态编码器来确定每个补丁的重要性分数，强调语义交织的音频-视频补丁。（2）重放引导的相关性评估：为了减少由于漂移而导致的先前学习的音频-视频知识的损坏，我们建议评估当前补丁在过去步骤上的相关性，以识别与过去步骤具有高相关性的补丁。基于这两个想法的结果，我们执行概率性补丁选择，以实现有效的持续音频-视频预训练。在多个基准上的实验验证表明，与强持续学习基线相比，我们的方法在零样本检索任务中实现了3.69个百分点的相对性能提升，同时将内存消耗减少了约45%。
英文摘要：
Continuously learning a variety of audio-video semantics over time is crucial for audio-related reasoning tasks in our ever-evolving world. However, this is a nontrivial problem and poses two critical challenges: sparse spatio-temporal correlation between audio-video pairs and multimodal correlation overwriting that forgets audio-video relations. To tackle this problem, we propose a new continual audio-video pre-training method with two novel ideas: (1) Localized Patch Importance Scoring: we introduce a multimodal encoder to determine the importance score for each patch, emphasizing semantically intertwined audio-video patches. (2) Replay-guided Correlation Assessment: to reduce the corruption of previously learned audiovisual knowledge due to drift, we propose to assess the correlation of the current patches on the past steps to identify the patches exhibiting high correlations with the past steps. Based on the results from the two ideas, we perform probabilistic patch selection for effective continual audio-video pre-training. Experimental validation on multiple benchmarks shows that our method achieves a 3.69%p of relative performance gain in zero-shot retrieval tasks compared to strong continual learning baselines, while reducing memory consumption by ~45%.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
