arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2508.11272cs.CVcs.AI

通过推理增强的表示工程提升监督性复合图像检索

Enhancing Supervised Composed Image Retrieval via Reasoning-Augmented Representation Engineering

Jun Li(李俊);Hongjian Dou(董宏健);Zhenyu Zhang(张振宇);Kai Li(李凯);Shaoguo Liu(刘少国);Tingting Gao(高婷婷)

Jun Li, Hongjian Dou, Zhenyu Zhang, Kai Li, Shaoguo Liu, Tingting Gao

AI总结:

本文提出PMTFR框架,通过推理增强的表示工程提升监督性复合图像检索性能,实验表明其优于现有方法。

摘要:

复合图像检索(CIR)提出了重大挑战,因为它需要联合理解参考图像和修改后的文本指令以找到相关的目标图像。一些现有方法尝试使用两阶段方法进一步细化检索结果。然而,这通常需要额外训练排名模型。尽管链式推理(CoT)技术在降低语言模型训练成本方面取得了成功,但其在CIR任务中的应用仍然有限--压缩视觉信息到文本或依赖于复杂的提示设计。此外,现有工作仅将其用于零样本CIR,因为使用良好训练的模型在监督CIR中实现满意结果具有挑战性。在本工作中,我们提出了一种框架,其中包括具有无训练细化的金字塔匹配模型(PMTFR)以解决这些挑战。通过一个简单但有效的模块称为金字塔修补器,我们增强了金字塔匹配模型对不同粒度的视觉信息的理解。受表示工程的启发,我们从CoT数据中提取表示并将其注入到LVLMs中。这种方法使我们能够在无训练细化范式下获得细化的检索分数,而无需依赖显式的文本推理,进一步提高了性能。在CIR基准上的广泛实验表明,PMTFR在监督CIR任务中超过了最先进的方法。代码将公开。

英文摘要:

Composed Image Retrieval (CIR) presents a significant challenge as it requires jointly understanding a reference image and a modified textual instruction to find relevant target images. Some existing methods attempt to use a two-stage approach to further refine retrieval results. However, this often requires additional training of a ranking model. Despite the success of Chain-of-Thought (CoT) techniques in reducing training costs for language models, their application in CIR tasks remains limited -- compressing visual information into text or relying on elaborate prompt designs. Besides, existing works only utilize it for zero-shot CIR, as it is challenging to achieve satisfactory results in supervised CIR with a well-trained model. In this work, we proposed a framework that includes the Pyramid Matching Model with Training-Free Refinement (PMTFR) to address these challenges. Through a simple but effective module called Pyramid Patcher, we enhanced the Pyramid Matching Model's understanding of visual information at different granularities. Inspired by representation engineering, we extracted representations from COT data and injected them into the LVLMs. This approach allowed us to obtain refined retrieval scores in the Training-Free Refinement paradigm without relying on explicit textual reasoning, further enhancing performance. Extensive experiments on CIR benchmarks demonstrate that PMTFR surpasses state-of-the-art methods in supervised CIR tasks. The code will be made public.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑