Conan-embedding-v3: 融合模态特定模型实现全模态嵌入
Conan-embedding-v3: Fusing Modality-Specific Models for Omni-Modal Embedding
AI总结：
提出解耦-融合-恢复框架，通过独立训练模态专家并融合任务向量，再使用投影器恢复和平衡多模态重演解决投影器漂移问题，实现单一骨干网络支持文本、图像、视频、文档和音频检索。
摘要：
全模态检索承诺为文本、图像、视频、文档和音频输入提供单一嵌入空间，但由于这些模态在数据分布、架构和优化动态上存在差异，构建这样一个统一的检索器十分困难。在这项工作中，我们提出了Conan-embedding-v3，一个用于全模态检索的解耦-融合-恢复框架。Conan-embedding-v3首先独立训练模态专家，然后将它们的任务向量融合到一个单一的密集骨干网络中，我们称这种策略为解耦专家融合。我们表明，这种融合组合了视觉、视频和文档检索能力，但也暴露了基于投影器的模态的一个失败模式：当通过外部编码器和投影器附加音频时，融合骨干网络会使投影器校准到音频专家骨干网络，导致尽管原封不动地复制了所有音频特定模块，音频检索性能仍大幅下降。我们将这种失败称为投影器漂移。为了修复它，Conan-embedding-v3应用了投影器恢复（即在保持骨干网络冻结的情况下对投影器进行全参数微调），随后进行平衡的多模态重演。得到的模型在一个骨干网络中支持这些检索路径，在MMEB上达到74.9分，同时在30任务的MAEB音频套件上获得55.61分。
英文摘要：
Omni-modal retrieval promises a single embedding space for text, image, video, document, and audio inputs, but building such a unified retriever is difficult since these modalities differ in data distribution, architecture, and optimization dynamics. In this work, we present Conan-embedding-v3, a decouple--fuse--recover framework for omni-modal retrieval. Conan-embedding-v3 first trains modality specialists independently and fuses their task vectors into a single dense backbone, a strategy we call Decoupled Specialist Fusion. We show that this fusion composes visual, video, and document retrieval capabilities, but also exposes a failure mode for projector-based modalities: when audio is attached through an external encoder and projector, fusing the backbone leaves the projector calibrated to the audio-specialist backbone, causing a large audio retrieval regression despite copying all audio-specific modules unchanged. We call this failure Projector Drift. To repair it, Conan-embedding-v3 applies Projector Recovery (i.e., full-parameter fine-tuning of the projector while keeping the backbone frozen) followed by balanced multi-modal rehearsal. The resulting model supports these retrieval pathways in one backbone, achieving 74.9 scores on MMEB while obtaining 55.61 on the 30-task MAEB audio suite.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
