The ability to efficiently search for images is essential for improving the user experiences across various products. Incorporating user feedback, via multi-modal inputs, to navigate visual search can help tailor retrieved results to specific user queries. We focus on the task of text-conditioned image retrieval that utilizes support text feedback alongside a reference image to retrieve images that concurrently satisfy constraints imposed by both inputs. The task is challenging since it requires learning composite image-text features by incorporating multiple cross-granular semantic edits from text feedback and then applying the same to visual features. To address this, we propose a novel framework SAC which resolves the above in two major steps: "where to see" (Semantic Feature Attention) and "how to change" (Semantic Feature Modification). We systematically show how our architecture streamlines the generation of text-aware image features by removing the need for various modules required by other state-of-art techniques. We present extensive quantitative, qualitative analysis, and ablation studies, to show that our architecture SAC outperforms existing techniques by achieving state-of-the-art performance on 3 benchmark datasets: FashionIQ, Shoes, and Birds-to-Words, while supporting natural language feedback of varying lengths.
The Visual Target Matters: Learning across the Visual Hierarchy for Brain-to-Image Retrieval
Chongqing University of Posts and Telecommunications(重庆邮电大学);Shanghai Jiaotong University(上海交通大学);Beijing University of Chemical Technology(北京化工大学);The Hong Kong Polytechnic University(香港理工大学)
Cross-Resolution Semantic Transfer for Robust Text-to-Image Retrieval in Low-Resolution Surveillance
School of Computer Science and Technology, Wuhan University of Science and Technology(武汉科技大学计算机科学与技术学院);School of Computer Science, National Engineering Research Center for Multimedia Software, Wuhan University(武汉大学计算机学院,国家多媒体软件工程技术研究中心);Hubei Province Key Laboratory of Intelligent Information Processing and Real-time Industrial System, Wuhan University of Science and Technology(湖北省智能信息处理与实时工业系统重点实验室,武汉科技大学)
University of California, Berkeley(加州大学伯克利分校);Tsinghua University(清华大学);ICSI, Berkeley(伯克利国际计算机科学研究所);National University of Singapore(新加坡国立大学);University of Massachusetts, Lowell(马萨诸塞大学洛厄尔分校)
Image Retrieval on Real-life Images with Pre-trained Vision-and-Language Models
Australian National University(澳大利亚国立大学);Australian Institute for Machine Learning(澳大利亚机器学习研究所);University of Adelaide(阿德莱德大学);Idiap Research Institute(伊迪亚普研究所)
State-Conditioned Visual Evidence Retrieval for Fine-Grained Perception in Document Vision-Language Models
College of Computer Science and Artificial Intelligence, Fudan University(复旦大学计算机科学与人工智能学院);Shanghai Innovation Institute(上海创新研究院);Shanghai Artificial Intelligence Laboratory(上海人工智能实验室);ByteDance, LarkAI(字节跳动 LarkAI);Shanghai Key Lab of Intelligent Information Processing(上海智能信息处理重点实验室)
Concentrate After Imagination: Text-Conditioned Evidence Grounding for Partially Relevant Video Retrieval
The Hong Kong University of Science and Technology (Guangzhou)(香港科技大学(广州));University of Electronic Science and Technology of China(电子科技大学);The Hong Kong University of Science and Technology(香港科技大学)
Retrieval-Augmented Anatomical Guidance for Text-to-CT Generation
Unit of Artificial Intelligence and Computer Systems, Department of Engineering, Università Campus Bio-Medico di Roma, Rome, Italy(人工智能与计算机系统单位,工程系,罗马生物医学大学,意大利罗马);Department of Diagnostics and Intervention, Biomedical Engineering and Radiation Physics, Umeå University, Umeå, Sweden(诊断与介入系,生物医学工程与放射物理,乌梅大学,瑞典乌梅)
Proximity-Based Evidence Retrieval for Uncertainty-Aware Neural Networks
Faculty of Engineering & IT, University of Technology Sydney(悉尼技术大学工程与信息学院);University Research and Innovation Center (EKIK), Óbuda University(奥布达大学研究与创新中心)
TEVI: Text-Conditioned Editing of Visual Representations via Sparse Autoencoders for Improved Vision-Language Alignment
Max Planck Institute for Informatics, Saarland Informatics Campus, Saarbrücken, Germany(马克斯·普朗克研究所信息学院,萨尔兰信息学院,德国萨尔布吕肯);Department of Language Science and Technology, Saarland University, Saarbrücken, Germany(语言科学与技术系,萨尔兰大学,德国萨尔布吕肯)
XGPT: Cross-modal Generative Pre-Training for Image Captioning
MOE Key Laboratory of Computational Linguistics, Peking University(北京大学计算语言学教育部重点实验室);Microsoft Research Asia(微软亚洲研究院);Microsoft(微软)
虽然许多基于BERT的跨模态预训练模型在图像-文本检索和VQA等下游理解任务中产生了出色的结果,但它们不能直接应用于生成任务。在本文中,我们提出了XGPT,一种新的跨模态生成预训练方法的图像描述,旨在通过三个新的生成任务,包括图像条件掩蔽语言建模(IMLM),图像条件去噪自动编码(IDA),和文本条件图像特征生成(TIFG)的文本到图像的字幕生成器进行预训练。因此,预先训练的XGPT可以在没有任何特定于任务的架构修改的情况下进行微调,以创建用于图像描述的最先进的模型。实验表明,XGPT在COCO Captions和Flickr 30 k Captions等基准数据集上获得了新的最新结果。我们还使用XGPT生成新的图像标题作为图像检索任务的数据增强,并在所有召回指标上实现了显着的改善。
Efficient Media Retrieval from Non-Cooperative Queries
University of Illinois at Urbana-Champaign(伊利诺伊大学厄巴纳-香槟分校);eBay Research Labs(eBay研究实验室)
文本在人工世界中无处不在,当涉及到书名和作者姓名时,文本很容易获得。使用来自Stanford Mobile Visual Search数据集的书籍封面集的图像以及来自openlibrary.org的其他书籍封面和元数据,我们构建了一个大规模的书籍封面检索数据集,每个数据集包含10万个干扰项封面以及标题和作者字符串。由于我们的查询图像是不好的条件,干净的文本提取,我们提出了一种方法,用于提取匹配的噪声和错误的OCR读数,并匹配它对干净的作者和书名字符串在一个标准的文档查找问题设置。最后,我们演示了如何使用这种文本匹配作为一个功能,结合流行的检索功能,如VLAD使用一个简单的学习设置,以实现检索精度的显着改善,无论是VLAD或单独的文本。