arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2603.04772cs.CLcs.AI

TSEmbed: 解锁通用多模态嵌入中的任务扩展

TSEmbed: Unlocking Task Scaling in Universal Multimodal Embeddings

State Key Laboratory of IOTSC, University of Macau(物联网科学与技术国家重点实验室,澳门大学);OPPO Research Institute(OPPO研究院)

Yebo Wu, Feng Liu, Ziwei Xie, Zhiyuan Liu, Changwang Zhang, Jun Wang, Li Li

AI总结:

TSEmbed通过融合MoE与LoRA,结合专家感知负采样策略,提升通用多模态嵌入的任务扩展能力,并在基准和工业数据集上取得最佳性能。

摘要:

尽管多模态大语言模型(MLLMs)表现出卓越的推理能力,但将其适应为通用嵌入模型受到任务冲突的显著阻碍。为此,我们提出了TSEmbed,一种通用多模态嵌入框架,通过混合专家(MoE)与低秩适应(LoRA)相结合,显式解耦冲突的任务目标。此外,我们引入了专家感知负采样(EANS),一种利用专家路由分布作为内在语义相似性代理的新策略。通过动态优先选择与查询共享专家激活模式的信息性困难负样本,EANS有效增强了模型的判别能力并细化嵌入边界。为确保训练稳定性,我们进一步设计了两阶段学习范式,在优化表示之前先巩固专家专业化。TSEmbed在大规模多模态嵌入基准(MMEB)和实际工业生产数据集上均取得最先进的性能,为通用多模态嵌入的任务级扩展奠定了基础。

英文摘要:

Despite the exceptional reasoning capabilities of Multimodal Large Language Models (MLLMs), their adaptation into universal embedding models is significantly impeded by task conflict. To address this, we propose TSEmbed, a universal multimodal embedding framework that synergizes Mixture-of-Experts (MoE) with Low-Rank Adaptation (LoRA) to explicitly disentangle conflicting task objectives. Moreover, we introduce Expert-Aware Negative Sampling (EANS), a novel strategy that leverages expert routing distributions as an intrinsic proxy for semantic similarity. By dynamically prioritizing informative hard negatives that share expert activation patterns with the query, EANS effectively sharpens the model's discriminative power and refines embedding boundaries. To ensure training stability, we further devise a two-stage learning paradigm that solidifies expert specialization before optimizing representations via EANS. TSEmbed achieves state-of-the-art performance on both the Massive Multimodal Embedding Benchmark (MMEB) and real-world industrial production datasets, laying a foundation for task-level scaling in universal multimodal embeddings.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑