arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2602.23369cs.IRcs.AIcs.CL
收藏

对比之理:一种级联多模态检索框架

Reason to Contrast: A Cascaded Multimodal Retrieval Framework

AI at Meta(Meta人工智能部门);University of Central Florida(中央佛罗里达大学)

Xuanming Cui, Hong-You Chen, Hao Yu, Hao Yuan, Zihao Wang, Shlok Kumar Mishra, Hanchao Yu, Yonghuan Yang, Jun Xiao, Ser-Nam Lim, Jianpeng Cheng, Qi Guo, Xiangjun Fan

AI总结:

本文提出TTE-v2框架,通过引入基于额外输入标记预算的推理驱动性能扩展,提升多模态检索的测试性能。

摘要:

传统多模态检索系统主要依赖于双编码架构,其中性能紧密依赖于嵌入维度。最近的工作,Think-Then-Embed(TTE),表明在嵌入之前加入多模态推理以提取额外的信息性标记可以进一步提高检索性能。在本文中,我们扩展了这一范式,提出TTE-v2,一种混合多模态检索框架,该框架引入了基于额外输入标记预算的推理驱动性能扩展,而不是模型或嵌入尺寸。我们的方法在初始多模态检索基础上增加额外的推理步骤用于重排序,使测试时查询-候选交互更具表现力。重排序阶段进一步为困难负样本挖掘和假负样本过滤提供细粒度监督,创建一个反馈循环,有效加强上游检索器。这种级联设计基于中间推理标记扩展在测试时带来显著的改进。在MMEB-V2基准测试中,TTE-v2-7B实现了新的最先进的准确率为75.7%,而TTE-v2-2B在使用显著更大的外部数据训练的领先7B模型上匹配或超越了它们。我们的结果突显了基于标记的扩展作为多模态检索替代扩展范式的潜力。

英文摘要:

Traditional multimodal retrieval systems rely primarily on bi-encoder architectures, where performance is closely tied to embedding dimensionality. Recent work, Think-Then-Embed (TTE), shows that incorporating multimodal reasoning to elicit additional informative tokens before embedding can further improve retrieval. In this paper, we extend this paradigm with TTE-v2, a hybrid multimodal retrieval framework that introduces reasoning-driven performance scaling based on additional input token budget rather than model or embedding size. Our approach augments the initial multimodal retrieval with additional reasoning steps for reranking, enabling more expressive query-candidate interactions at test time. The reranking stage further provides fine-grained supervision for hard negative mining and false negative filtering, creating a feedback loop that effectively strengthens the upstream retriever. This cascaded design delivers substantial test-time improvements based on intermediate reasoning token scaling. Experiments on the MMEB-V2 benchmark demonstrate that TTE-v2-7B achieves a new state-of-the-art accuracy of 75.7%, and that TTE-v2-2B matches or surpasses leading 7B models trained with significantly larger external data. Our results highlight the promise of token-wise scaling as an alternative scaling paradigm for multimodal retrieval.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑