Despite the evolution of deep-learning-based visual-textual processing systems, precise multi-modal matching remains a challenging task. In this work, we tackle the task of cross-modal retrieval through image-sentence matching based on word-region alignments, using supervision only at the global image-sentence level. Specifically, we present a novel approach called Transformer Encoder Reasoning and Alignment Network (TERAN). TERAN enforces a fine-grained match between the underlying components of images and sentences, i.e., image regions and words, respectively, in order to preserve the informative richness of both modalities. TERAN obtains state-of-the-art results on the image retrieval task on both MS-COCO and Flickr30k datasets. Moreover, on MS-COCO, it also outperforms current approaches on the sentence retrieval task.
Focusing on scalable cross-modal information retrieval, TERAN is designed to keep the visual and textual data pipelines well separated. Cross-attention links invalidate any chance to separately extract visual and textual features needed for the online search and the offline indexing steps in large-scale retrieval systems. In this respect, TERAN merges the information from the two domains only during the final alignment phase, immediately before the loss computation. We argue that the fine-grained alignments produced by TERAN pave the way towards the research for effective and efficient methods for large-scale cross-modal information retrieval. We compare the effectiveness of our approach against relevant state-of-the-art methods. On the MS-COCO 1K test set, we obtain an improvement of 5.7% and 3.5% respectively on the image and the sentence retrieval tasks on the Recall@1 metric. The code used for the experiments is publicly available on GitHub at https://github.com/mesnico/TERAN.
Fine-Grained Scene Image Classification with Modality-Agnostic Adapter
School of Computer Science, Fudan University（复旦大学计算机科学学院）；Innovation Center of Calligraphy and Painting Creation Technology, MCT（中国传媒书画创作技术创新中心）；Videt Technology（维德科技）
Dual-path CNN with Max Gated block for Text-Based Person Re-identification
Nanjing University of information science and Technology（南京信息工程大学）；Xinjiang University（新疆大学）；Nanjing Institute of Technology（南京工程学院）；KingSaud University（沙特国王大学）
Cross-modal Subspace Learning for Fine-grained Sketch-based Image Retrieval
Beijing University of Posts and Telecommunications（北京邮电大学）；Chinese Academy of Sciences（中国科学院）；Queen Mary University of London（伦敦玛丽女王大学）；Victoria University of Wellington（惠灵顿维多利亚大学）
ShapeSpeak: Body Shape-Aware Textual Alignment for Visible-Infrared Person Re-Identification
Nanjing University of Science and Technology（南京理工大学）；Chongqing University of Posts and Telecommunications（重庆邮电大学）；The Hong Kong Polytechnic University（香港理工大学）
NEXT: Multi-Grained Mixture of Experts via Text-Modulation for Multi-Modal Object Re-Identification
State Key Laboratory of Opto-Electronic Information Acquisition and Protection Technology（光电信息采集与防护技术国家重点实验室）；Anhui Provincial Key Laboratory of Multimodal Cognitive Computation（安徽省多模态认知计算重点实验室）；School of Artificial Intelligence（人工智能学院）；Anhui University（安徽大学）；State Key Laboratory of Multimodal Artificial Intelligence Systems（多模态人工智能系统国家重点实验室）；New Laboratory of Pattern Recognition（模式识别新实验室）；CASIA（中国科学院自动化所）；University of Chinese Academy of Sciences（中国科学院大学）；School of Computing and Mathematical Sciences（计算与数学科学学院）；University of Greenwich（格林威治大学）
Cross-Modal Fusion Distillation for Fine-Grained Sketch-Based Image Retrieval
University of Exeter（埃克塞特大学）；University of Tübingen（图宾根大学）；MPI for Informatics（马克斯·普朗克信息学研究所）；MPI for Intelligent Systems（马克斯·普朗克智能系统研究所）；University of Surrey（萨里大学）
Self-paced Multi-grained Cross-modal Interaction Modeling for Referring Expression Comprehension
School of Software Technology, Zhejiang University（浙江大学软件学院）；College of Computer Science and Technology, Zhejiang University（浙江大学计算机科学与技术学院）；Zhejiang University-University of Illinois at Urbana-Champaign Institute, Zhejiang University（浙江大学-伊利诺伊大学厄巴纳-香槟分校研究院）
CoMLP: Cooperatively-Gated MLPs for Fine-Grained Cross-Modal Information Fusion in Medical Image Segmentation
Institute of Translational Medicine, Shanghai Jiao Tong University（上海交通大学转化医学研究院）；Zhongguancun Academy（中关村学院）；Zhongguancun Institute of Artificial Intelligence（中关村人工智能研究院）；School of Computer Science, the University of Sydney（悉尼大学计算机学院）；China United Network Communications Group Co., Ltd.（中国联合网络通信集团有限公司）
Unveil: Unified Visual-Textual Integration and Distillation for Multi-modal Document Retrieval
State Key Laboratory of General Artificial Intelligence, Peking University（北京理工大学通用人工智能国家重点实验室）；School of Intelligence Science and Technology, Peking University（北京大学智能科学与技术学院）；Aerospace Information Research Institute, Chinese Academy of Sciences（中国科学院航空航天信息研究所）；Key Laboratory of Target Cognition and Application Technology（目标认知与应用技术重点实验室）；Beijing Institute of Technology（北京理工大学）；Ucap Cloud（Ucap云）
VISTANet: VIsual Spoken Textual Additive Net for Interpretable Multimodal Emotion Recognition
Center for Machine Vision and Signal Analysis, University of Oulu（机器视觉与信号分析中心，奥卢大学）；Indian Institute of Technology Roorkee（印度理工学院罗尔基分校）；State Key Laboratory of Blockchain and Data Security, Zhejiang University（区块链与数据安全国家重点实验室，浙江大学）
MFCLIP: Multi-modal Fine-grained CLIP for Generalizable Diffusion Face Forgery Detection
Faculty of Computer Science and Technology, Qilu University of Technology (Shandong Academy of Sciences)（计算机科学与技术学院，齐鲁工业大学（山东科学院））；School of Computing, National University of Singapore（computing 学院，新加坡国立大学）；School of Computing and Information Technology, Great Bay University（computing 与信息学院，大湾大学）；National Engineering Laboratory for Big Data System Computing Technology, Shenzhen University（大数据系统计算技术国家工程实验室，深圳大学）
Multi-Modal Motion Retrieval by Learning a Fine-Grained Joint Embedding Space
Southern University of Science and Technology（南方科技大学）；Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences（深圳先进技术研究院，中国科学院）；University of Chinese Academy of Sciences（中国科学院大学）；ShanghaiTech University（上海科技大学）；Nanyang Technological University（南洋理工大学）；Faculty of Computer Science and Control Engineering, Shenzhen University of Advanced Technology（计算机科学与控制工程学院，深圳大学先进技术学院）
