CLIP-Branches: 面向文本-图像检索的交互式微调
CLIP-Branches: Interactive Fine-Tuning for Text-Image Retrieval
AI总结:
针对CLIP类文本-图像检索结果精度与完整性不足的问题,提出CLIP-Branches搜索引擎,通过交互式微调让用户迭代标注正负例优化查询,结合预构建高效索引保证响应速度,提升检索相关性与准确性。
摘要:
文本-图像模型(最具代表性的是CLIP)的出现,极大地改变了信息检索的格局。这类模型能够融合文本、图像等多种模态。CLIP的一项重要成果是支持用户以文本为查询词搜索图像,反之亦然,这是通过图像与文本数据的联合嵌入实现的,该嵌入可用于检索相似条目等场景。尽管已有近似最近邻搜索等高效查询处理技术,但检索结果仍可能存在精度不足、完整性欠缺的问题。我们提出了CLIP-Branches,这是一种基于CLIP架构的新型文本-图像搜索引擎。我们的方法通过加入交互式微调阶段对传统文本-图像搜索引擎进行增强,该阶段允许用户通过迭代定义正例和负例来进一步明确搜索查询。我们的框架会基于额外的用户反馈训练一个分类模型,本质上输出整个数据目录中所有被分类为正例的实例。不过,依托最新技术,该推理阶段并非通过扫描整个数据目录实现,而是采用为数据预构建的高效索引结构。我们的实验结果表明,微调后的结果能够在保持快速响应时间的同时,提升初始搜索输出的相关性和准确性。
英文摘要:
The advent of text-image models, most notably CLIP, has significantly transformed the landscape of information retrieval. These models enable the fusion of various modalities, such as text and images. One significant outcome of CLIP is its capability to allow users to search for images using text as a query, as well as vice versa. This is achieved via a joint embedding of images and text data that can, for instance, be used to search for similar items. Despite efficient query processing techniques such as approximate nearest neighbor search, the results may lack precision and completeness. We introduce CLIP-Branches, a novel text-image search engine built upon the CLIP architecture. Our approach enhances traditional text-image search engines by incorporating an interactive fine-tuning phase, which allows the user to further concretize the search query by iteratively defining positive and negative examples. Our framework involves training a classification model given the additional user feedback and essentially outputs all positively classified instances of the entire data catalog. By building upon recent techniques, this inference phase, however, is not implemented by scanning the entire data catalog, but by employing efficient index structures pre-built for the data. Our results show that the fine-tuned results can improve the initial search outputs in terms of relevance and accuracy while maintaining swift response times
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。