arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2009.01485cs.CVcs.AI
收藏

SAC:文本条件下图像检索的语义注意组合

SAC: Semantic Attention Composition for Text-Conditioned Image Retrieval

Adobe(奥多比公司);Microsoft(微软公司);Indian Institute of Technology, Kharagpur(印度理工学院卡尔哈格普尔分校);Massachusetts Institute of Technology(麻省理工学院)

Surgan Jandial, Pinkesh Badjatiya, Pranit Chawla, Ayush Chopra, Mausoom Sarkar, Balaji Krishnamurthy

摘要:

高效搜索图像的能力对于改善各种产品的用户体验至关重要。通过多模态输入来验证用户反馈以导航视觉搜索可以帮助针对特定用户查询定制检索结果。我们专注于文本条件图像检索的任务,利用支持文本反馈旁边的参考图像检索图像,同时满足两个输入的约束。这项任务是具有挑战性的,因为它需要通过从文本反馈中整合多个跨粒度语义编辑来学习复合图像-文本特征,然后将其应用于视觉特征。为了解决这个问题,我们提出了一个新的框架SAC解决上述两个主要步骤:“在哪里看到”(语义特征注意)和“如何改变”(语义特征修改)。我们系统地展示了我们的架构如何通过消除对其他最先进技术所需的各种模块的需要来简化文本感知图像功能的生成。我们提出了广泛的定量,定性分析和消融研究,以表明我们的架构SAC通过在3个基准数据集上实现最先进的性能而优于现有技术:FashionIQ,Shoes和Birds-to-Words,同时支持不同长度的自然语言反馈。

英文摘要:

The ability to efficiently search for images is essential for improving the user experiences across various products. Incorporating user feedback, via multi-modal inputs, to navigate visual search can help tailor retrieved results to specific user queries. We focus on the task of text-conditioned image retrieval that utilizes support text feedback alongside a reference image to retrieve images that concurrently satisfy constraints imposed by both inputs. The task is challenging since it requires learning composite image-text features by incorporating multiple cross-granular semantic edits from text feedback and then applying the same to visual features. To address this, we propose a novel framework SAC which resolves the above in two major steps: "where to see" (Semantic Feature Attention) and "how to change" (Semantic Feature Modification). We systematically show how our architecture streamlines the generation of text-aware image features by removing the need for various modules required by other state-of-art techniques. We present extensive quantitative, qualitative analysis, and ablation studies, to show that our architecture SAC outperforms existing techniques by achieving state-of-the-art performance on 3 benchmark datasets: FashionIQ, Shoes, and Birds-to-Words, while supporting natural language feedback of varying lengths.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑