Recognition and retrieval of textual content from the large document collections have been a powerful use case for the document image analysis community. Often the word is the basic unit for recognition as well as retrieval. Systems that rely only on the text recogniser (OCR) output are not robust enough in many situations, especially when the word recognition rates are poor, as in the case of historic documents or digital libraries. An alternative has been word spotting based methods that retrieve/match words based on a holistic representation of the word. In this paper, we fuse the noisy output of text recogniser with a deep embeddings representation derived out of the entire word. We use average and max fusion for improving the ranked results in the case of retrieval. We validate our methods on a collection of Hindi documents. We improve word recognition rate by 1.4 and retrieval by 11.13 in the mAP.
University of Insubria（因苏布里亚大学）；National University of Sciences and Technology（国立科技大学）
This paper proposes a cross-modal retrieval system that leverages on image and text encoding. Most multimodal architectures employ separate networks for each modality to capture the semantic relationship between them. However, in our work image-text encoding can achieve comparable results in terms of cross-modal retrieval without having to use a separate network for each modality. We show that text encodings can capture semantic relationships between multiple modalities. In our knowledge, this work is the first of its kind in terms of employing a single network and fused image-text embedding f…
SPARK-IL: Spectral Retrieval-Augmented RAG for Knowledge-driven Deepfake Detection via Incremental Learning
Univ. Polytechnique Hauts-de-France（上法兰西理工大学）；Institute of Applied Sciences and Intelligent Systems – Lecce（应用科学与智能系统研究所（莱切））；Department of Innovation Engineering, University of Salento（萨伦托大学创新工程系）；Sorbonne Center for Artificial Intelligence, Sorbonne University Abu Dhabi（索邦大学阿布扎比分校索邦人工智能中心）
Amazon（亚马逊）；The University of Adelaide（阿德莱德大学）；University of New South Wales（新南威尔士大学）；Zhejiang University（浙江大学）
We introduce Retrieval Augmented Classification (RAC), a generic approach to augmenting standard image classification pipelines with an explicit retrieval module. RAC consists of a standard base image encoder fused with a parallel retrieval branch that queries a non-parametric external memory of pre-encoded images and associated text snippets. We apply RAC to the problem of long-tail classification and demonstrate a significant improvement over previous state-of-the-art on Places365-LT and iNaturalist-2018 (14.5% and 6.7% respectively), despite using only the training datasets themselves as th…
Proximity-Based Evidence Retrieval for Uncertainty-Aware Neural Networks
Faculty of Engineering & IT, University of Technology Sydney（悉尼技术大学工程与信息学院）；University Research and Innovation Center (EKIK), Óbuda University（奥布达大学研究与创新中心）
The Chinese University of Hong Kong（香港中文大学）；SenseTime Research（商汤科技研究院）；Beihang University（北京航空航天大学）
Text-image cross-modal retrieval is a challenging task in the field of language and vision. Most previous approaches independently embed images and sentences into a joint embedding space and compare their similarities. However, previous approaches rarely explore the interactions between images and sentences before calculating similarities in the joint space. Intuitively, when matching between images and sentences, human beings would alternatively attend to regions in images and words in sentences, and select the most salient information considering the interaction between both modalities. In t…
The College of Information Engineering, Xiangtan University（湘潭大学信息工程学院）
Rapid increase of digitized document give birth to high demand of document image retrieval. While conventional document image retrieval approaches depend on complex OCR-based text recognition and text similarity detection, this paper proposes a new content-based approach, in which more attention is paid to features extraction and fusion. In the proposed approach, multiple features of document images are extracted by different CNN models. After that, the extracted CNN features are reduced and fused into weighted average feature. Finally, the document images are ranked based on feature similarit…
Visual Late Chunking: An Empirical Study of Contextual Chunking for Efficient Visual Document Retrieval
Hong Kong University of Science and Technology (Guangzhou)（香港科技大学（广州））；Alibaba Cloud Computing（阿里云）；Hong Kong University of Science and Technology（香港科技大学）
Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；Donghua University（东华大学）；Tencent（腾讯）
Videos contain multi-modal content, and exploring multi-level cross-modal interactions with natural language queries can provide great prominence to text-video retrieval task (TVR). However, new trending methods applying large-scale pre-trained model CLIP for TVR do not focus on multi-modal cues in videos. Furthermore, the traditional methods simply concatenating multi-modal features do not exploit fine-grained cross-modal information in videos. In this paper, we propose a multi-level multi-modal hybrid fusion (M2HF) network to explore comprehensive interactions between text queries and each m…
Despite recent advances in multi-scale deep representations, their limitations are attributed to expensive parameters and weak fusion modules. Hence, we propose an efficient approach to fuse multi-scale deep representations, called convolutional fusion networks (CFN). Owing to using 1$\times$1 convolution and global average pooling, CFN can efficiently generate the side branches while adding few parameters. In addition, we present a locally-connected fusion module, which can learn adaptive weights for the side branches and form a discriminatively fused feature. CFN models trained on the CIFAR …
Imagine Before Concentration: Diffusion-Guided Registers Enhance Partially Relevant Video Retrieval
Harbin Institute of Technology, Shenzhen（哈尔滨工业大学（深圳））；Tsinghua Shenzhen International Graduate School, Tsinghua University（清华大学深圳国际研究生院）；Peng Cheng Laboratory（鹏城实验室）
GOMA: Toward Structure-Driven Multimodal Alignment from a Graph Signal Smoothing Perspective
School of Airspace Science and Engineering, Shandong University（山东大学 airspace 科学与工程学院）；Department of Computer Science, Beijing Institute of Technology（北京理工大学计算机学院）；School of Computer Science and Engineering, Sun Yat-sen University（中山大学计算机科学与工程学院）
RefAtomNet++: Advancing Referring Atomic Video Action Recognition using Multi-Trajectory Semantic Retrieval
Institute for Anthropomatics and Robotics, Karlsruhe Institute of Technology（人机化研究所，卡尔斯鲁厄技术大学）；RISE Research Institutes of Sweden（瑞典RISE研究机构）；KTH Royal Institute of Technology（皇家理工学院）；School of Artificial Intelligence and Robotics（人工智能与机器人学院）；National Engineering Research Center of Robot Visual Perception and Control Technology（机器人视觉感知与控制技术国家工程研究中心）；Chinese University of Hong Kong（香港中文大学）；Shanghai AI Lab（上海人工智能实验室）
