图像搜索的当代艺术：通过视觉-语言模型实现迭代式用户意图扩展
The Contemporary Art of Image Search: Iterative User Intent Expansion via Vision-Language Model
AI总结：
本文提出基于视觉-语言模型的用户意图扩展框架，通过语言解析、视觉解析和逻辑组合两阶段过程，并结合迭代式上下文交互，显著提升NFT图像搜索的准确性与用户体验。
摘要：
图像搜索是探索海量数字图像库的一种必不可少且用户友好的方法。然而，现有的图像搜索方法严重依赖标签匹配或图像相似度等邻近性度量，需要用户提供精确的输入才能获得令人满意的结果。为了满足对能够准确理解用户搜索意图的当代图像搜索引擎日益增长的需求，我们引入了一个创新的用户意图扩展框架。我们的框架利用视觉-语言模型来解析和组合多模态用户输入，从而提供更准确、更令人满意的结果。该框架包含两个阶段的过程：1）解析阶段，该阶段包含一个与大型语言模型相结合的语言解析模块，以增强对文本输入的理解，同时还包含一个视觉解析模块，该模块集成了交互式分割模块，以快速识别图像中的详细视觉元素；2）逻辑组合阶段，该阶段将多个用户搜索意图组合成一个统一的逻辑表达式，以便在复杂的搜索场景中进行更精细的操作。此外，该意图扩展框架使用户能够与搜索结果进行灵活的上下文交互，从而迭代地进一步明确或调整其详细的搜索意图。我们将该框架实现到一个用于NFT（非同质化代币）搜索的图像搜索系统中，并进行了一项用户研究，以评估其可用性和新颖特性。结果表明，所提出的框架显著改善了用户的图像搜索体验。特别是解析和上下文交互被证明在帮助用户更准确地表达其搜索意图以及获得更愉快的迭代搜索体验方面非常有用。
英文摘要：
Image search is an essential and user-friendly method to explore vast galleries of digital images. However, existing image search methods heavily rely on proximity measurements like tag matching or image similarity, requiring precise user inputs for satisfactory results. To meet the growing demand for a contemporary image search engine that enables accurate comprehension of users' search intentions, we introduce an innovative user intent expansion framework. Our framework leverages visual-language models to parse and compose multi-modal user inputs to provide more accurate and satisfying results. It comprises two-stage processes: 1) a parsing stage that incorporates a language parsing module with large language models to enhance the comprehension of textual inputs, along with a visual parsing module that integrates an interactive segmentation module to swiftly identify detailed visual elements within images; and 2) a logic composition stage that combines multiple user search intents into a unified logic expression for more sophisticated operations in complex searching scenarios. Moreover, the intent expansion framework enables users to perform flexible contextualized interactions with the search results to further specify or adjust their detailed search intents iteratively. We implemented the framework into an image search system for NFT (non-fungible token) search and conducted a user study to evaluate its usability and novel properties. The results indicate that the proposed framework significantly improves users' image search experience. Particularly the parsing and contextualized interactions prove useful in allowing users to express their search intents more accurately and engage in a more enjoyable iterative search experience.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
