Image retrieval is crucial in robotics and computer vision, with downstream applications in robot place recognition and vision-based product recommendations. Modern retrieval systems face two key challenges: scalability and efficiency. State-of-the-art image retrieval systems train specific neural networks for each dataset, an approach that lacks scalability. Furthermore, since retrieval speed is directly proportional to embedding size, existing systems that use large embeddings lack efficiency. To tackle scalability, recent works propose using off-the-shelf foundation models. However, these models, though applicable across datasets, fall short in achieving performance comparable to that of dataset-specific models. Our key observation is that, while foundation models capture necessary subtleties for effective retrieval, the underlying distribution of their embedding space can negatively impact cosine similarity searches. We introduce Autoencoders with Strong Variance Constraints (AE-SVC), which, when used for projection, significantly improves the performance of foundation models. We provide an in-depth theoretical analysis of AE-SVC. Addressing efficiency, we introduce Single-shot Similarity Space Distillation ((SS)$_2$D), a novel approach to learn embeddings with adaptive sizes that offers a better trade-off between size and performance. We conducted extensive experiments on four retrieval datasets, including Stanford Online Products (SoP) and Pittsburgh30k, using four different off-the-shelf foundation models, including DinoV2 and CLIP. AE-SVC demonstrates up to a $16\%$ improvement in retrieval performance, while (SS)$_2$D shows a further $10\%$ improvement for smaller embedding sizes.
Exploring Auxiliary Context: Discrete Semantic Transfer Hashing for Scalable Image Retrieval
无监督哈希算法具有语义标签无关性、内存和搜索效率等优点,可以很好地支持基于内容的可伸缩图像检索。然而,由于图像表示的内在限制,学习的哈希码嵌入有限的判别语义。为了解决这个问题,在本文中,我们提出了一种新颖的哈希方法,称为\emph{Discrete Semantic Transfer Hashing}(DSTH)。其核心思想是\emph{directly}通过探索辅助上下文模态来增强离散图像哈希码的语义。为此,制定了一个统一的散列框架,同时保留图像的视觉相似性,并执行上下文模态的语义转移。此外,为了保证直接的语义传递和避免信息丢失,我们显式地施加离散约束,位不相关约束和位平衡约束的哈希码。提出了一种新颖有效的基于增广拉格朗日乘子的离散优化方法,用于迭代求解优化问题。整个学习过程具有线性的计算复杂度和良好的可扩展性。在三个基准数据集上的实验证明了DSTH与几种最先进的方法相比的优越性。
Exploiting Semantic Role Contextualized Video Features for Multi-Instance Text-Video Retrieval EPIC-KITCHENS-100 Multi-Instance Retrieval Challenge 2022
Institute for Infocomm Research, A*STAR(新加坡科技研究局资讯通信研究院);School of Computer Science and Engineering, NTU(南洋理工大学计算机科学与工程学院)
A Fuzzy Logic based Method for Efficient Retrieval of Vague and Uncertain Spatial Expressions in Text Exploiting the Granulation of the Spatial Event Queries
A Knowledge Graph and a Tripartite Evaluation Framework Make Retrieval-Augmented Generation Scalable and Transparent
Centre of Excellence for Data Science, Artificial Intelligence, and Modelling (DAIM), and the School of Computer Science(赫尔大学数据科学、人工智能与建模卓越中心及计算机科学学院);University of Hull
Histopathology WSI Encoding based on GCNs for Scalable and Efficient Retrieval of Diagnostically Relevant Regions
Beijing Advanced Innovation Center for Biomedical Engineering, Beihang University(北京航空航天大学生物医学工程北京高级创新中心);School of Software, Hefei University of Technology(合肥工业大学软件学院);Tianjin Institute of Industrial Biotechnology, Chinese Academy of Sciences(中国科学院天津工业生物技术研究所)
Beyond Retrieval: Learning Compact User Representations for Scalable LLM Personalization
Microsoft(微软公司);Shanghai International Studies University(上海国际问题研究大学);Zhejiang University(浙江大学);Department of Data Science and Artificial Intelligence, The Hong Kong Polytechnic University(数据科学与人工智能系,香港理工大学)
Accurate and Scalable Multimodal Pathology Retrieval via Attentive Vision-Language Alignment
Department of Computer Science and Engineering, The Hong Kong University of Science and Technology(香港科学与技术大学计算机科学与工程系);College of Computer Science and Technology, Zhejiang University(浙江大学计算机科学与技术学院);Department of Radiology, Union Hospital, Tongji Medical College, Huazhong University of Science and Technology(华中科技大学同济医学院附属同济医院放射科);Department of Pathology, Sir Run Run Shaw Hospital, School of Medicine, Zhejiang University(浙江大学医学院附属邵氏医院病理科);Department of Pathology, The Central Hospital of Wuhan, Tongji Medical College, Huazhong University of Science and Technology(华中科技大学同济医学院附属武汉中心医院病理科);Department of Pathology, The First Affiliated Hospital, School of Medicine, Zhejiang University(浙江大学医学院附属第一医院病理科);Department of Chemical and Biological Engineering, The Hong Kong University of Science and Technology(香港科学与技术大学化学与生物工程系);Division of Life Science, The Hong Kong University of Science and Technology(香港科学与技术大学生命科学系)
SignRAG: A Retrieval-Augmented System for Scalable Zero-Shot Road Sign Recognition
Computer Engineering The Ohio State University(计算机工程 俄亥俄州立大学);Applied Research Transportation Research Center Inc.(应用研究 交通运输研究中心公司);Computer Engineering The Ohio State University Columbus, Ohio, USA(计算机工程 俄亥俄州立大学 哥伦布俄亥俄州);Applied Research Transportation Research Center Inc. East Liberty, Ohio, USA(应用研究 交通运输研究中心公司 埃斯特利比 俄亥俄州)
AgriLens: Semantic Retrieval in Agricultural Texts Using Topic Modeling and Language Models
3 Dept. of Computer Science \& Applications, Sharda University, Greater Noida, Uttar Pradesh, India;4 The Chinese University of Hong Kong, Hong Kong SAR, China
Progressively Optimized Bi-Granular Document Representation for Scalable Embedding Based Retrieval
Beijing University of Posts and Telecommunications(北京邮电大学);Microsoft Research Asia(微软亚洲研究院);Microsoft STCA(微软STCA);University of Science and Technology of China(中国科学技术大学)
Optimizing Retrieval-Augmented Generation of Medical Content for Spaced Repetition Learning
Poznań University of Medical Sciences(波兹南医科大学);Adam Mickiewicz University(亚当·密茨凯维奇大学);Wydawnictwo Naukowe PWN(PWN科学出版社);SuperMemo World(SuperMemo世界公司);Association for Research and Applications of Artificial Intelligence(人工智能研究与应用协会)