OmniRetriever: 通过融合作为教师蒸馏实现任意到任意的音频-视频-文本检索
OmniRetriever: Any-to-Any Audio-Video-Text Retrieval via Fusion-as-Teacher Distillation
AI总结：
提出融合作为教师蒸馏方法，利用三元组嵌入的融合信号训练单模态嵌入，并构建OmniRetriever-7B模型，在零样本检索基准上超越现有方法，同时发布OmniRetriever-Bench基准。
摘要：
统一的多模态嵌入空间已成为跨模态检索和多模态RAG的标准接口，最近的音频-视频-文本（AVT）编码器将这一设置扩展到三种模态。当所有三种模态都可用时，此类编码器可以生成联合的（T,V,A）嵌入，但标准的成对InfoNCE目标在训练过程中未使用这一信号。我们通过融合作为教师蒸馏来弥补这一差距，将融合嵌入的停止梯度副本视为单模态嵌入的教师信号，并配以Tuple-InfoNCE项直接监督融合嵌入。我们将这一目标实例化为OmniRetriever-7B。在六个零样本检索基准上，OmniRetriever-7B在Clotho和SoundDescs上以R@1超过闭源的Gemini Embedding 2达13.3-18.0，并在MSR-VTT和MSVD上达到当代零样本专家级开放视频-文本编码器的水平。为了压力测试联合表示，我们进一步发布了OmniRetriever-Bench，这是一个包含12个方向的AVT检索基准，总计3782个三元组；在此基准上，OmniRetriever-7B达到AVG-all 34.84，比Gemini Embedding 2提高1.72，比之前最好的开源AVT方法提高8.03。
英文摘要：
Unified multimodal embedding spaces have become the standard interface for cross-modal retrieval and multimodal RAG, and recent audio-video-text (AVT) encoders extend this setting to three modalities. Such encoders can produce a joint (T,V,A) embedding whenever all three modalities are available, but standard pairwise InfoNCE objectives leave this signal unused during training. We close this gap with fusion-as-teacher distillation, which treats a stop-gradient copy of the fused embedding as a teacher signal for the single-modal embeddings, paired with a Tuple-InfoNCE term that supervises the fused embedding directly. We instantiate this objective as OmniRetriever-7B. Across six zero-shot retrieval benchmarks, OmniRetriever-7B surpasses the closed-source Gemini Embedding 2 by 13.3-18.0 R@1 on Clotho and SoundDescs, and reaches the contemporary zero-shot specialist band of open video-text encoders on MSR-VTT and MSVD. To stress-test joint representations, we further release OmniRetriever-Bench, a 12-direction AVT retrieval benchmark totaling 3782 triples; on it OmniRetriever-7B attains AVG-all 34.84, improving over Gemini Embedding 2 by 1.72 and over the best prior open-source AVT method by 8.03.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
