Dual encoders and cross encoders have been widely used for image-text retrieval. Between the two, the dual encoder encodes the image and text independently followed by a dot product, while the cross encoder jointly feeds image and text as the input and performs dense multi-modal fusion. These two architectures are typically modeled separately without interaction. In this work, we propose LoopITR, which combines them in the same network for joint learning. Specifically, we let the dual encoder provide hard negatives to the cross encoder, and use the more discriminative cross encoder to distill its predictions back to the dual encoder. Both steps are efficiently performed together in the same model. Our work centers on empirical analyses of this combined architecture, putting the main focus on the design of the distillation objective. Our experimental results highlight the benefits of training the two encoders in the same network, and demonstrate that distillation can be quite effective with just a few hard negative examples. Experiments on two standard datasets (Flickr30K and COCO) show our approach achieves state-of-the-art dual encoder performance when compared with approaches using a similar amount of data.
SEA: Sentence Encoder Assembly for Video Retrieval by Textual Queries
Renmin University of China(中国人民大学)
通过文本查询来检索无标签视频,称为Ad-hoc Video Search(AVS),是多媒体数据管理和检索的核心主题。AVS的成功依赖于跨模态表示学习,该学习将查询语句和视频编码到公共空间中以进行语义相似度计算。受先前少数作品在结合多个句子编码器方面取得的初步成功的启发,本文通过开发一种新的通用方法来有效地利用不同的句子编码器,向前迈出了一步。所提出的方法,我们术语句子编码器组件(SEA)的新颖性,是双重的。首先,与仅使用单个公共空间的现有技术不同,SEA支持多个编码器特定的公共空间中的文本-视频匹配。这样的属性防止匹配被产生比其他编码器长得多的编码向量的特定编码器支配。其次,为了探索个体公共空间之间的互补性,我们提出了多空间多损失学习。在MSR-VTT、TRECVID AVS 2016-2019、TGIF和MSVD四个基准测试上的大量实验表明,SEA超越了最先进的水平。此外,SEA非常容易实现。所有这些都使SEA成为AVS的一个有吸引力的解决方案,并有希望通过收获新的句子编码器来不断推进任务。
How to Make Cross Encoder a Good Teacher for Efficient Image-Text Retrieval?
State Key Laboratory of Multimodal Artificial Intelligence Systems(多模态人工智能系统国家重点实验室);School of Information Science and Technology, ShanghaiTech University(上海科技大学信息科学与技术学院);The University of Hong Kong(香港大学);School of Artificial Intelligence, University of Chinese Academy of Sciences(中国科学院大学人工智能学院)
Category-level Text-to-Image Retrieval Improved: Bridging the Domain Gap with Diffusion Models and Vision Encoders
King Abdullah University of Science and Technology(国王阿卜杜勒-阿齐兹大学科学与技术大学);Visual Recognition Group(视觉识别组);Faculty of Electrical Engineering(电气工程学院);Czech Technical University in Prague(布拉格捷克技术大学)
SEDS: Semantically Enhanced Dual-Stream Encoder for Sign Language Retrieval
University of Science and Technology of China(中国科学技术大学);Hefei Comprehensive National Science Center(合肥综合性国家科学中心);Merchants Union Consumer Finance Company Limited(招商局消费金融有限公司)
MILES: Visual BERT Pre-training with Injected Language Semantics for Video-text Retrieval
The University of Hong Kong(香港大学);ARC Lab(ARC实验室);UC Berkeley(加州大学伯克利分校);National University of Singapore(新加坡国立大学);Tsinghua University(清华大学);Tencent PCG(腾讯平台与内容事业群)
FusionBERT: Multi-View Image--3D Retrieval via Cross-Attention Visual Fusion and Normal-Aware 3D Encoder
IROOTECH TECHNOLOGY;Wolf 1069 b Lab, Sany Group(三一集团Wolf 1069 b实验室);Zhejiang University(浙江大学);School of Computer Science and Engineering, Central South University(中南大学计算机科学与工程学院);BPIT, Sany Group(三一集团BPIT)
Recognizing Families through Images with Pretrained Encoder
Faculty of Information Technology, University of Science, VNU-HCM(越南国家大学胡志明市分校科学大学信息技术学院)
亲属关系验证和亲属关系检索是计算机视觉中的新兴任务。亲属关系验证的目的是确定两个人脸图像是否来自相关的人,而亲属关系检索是从图像库中检索可能与一个人相关的人脸图像的任务。他们引入了独特的挑战,因为隐藏的关系和功能,进行面部图像之间的固有特性。我们采用3种方法,FaceNet,Siamese VGG-Face以及FaceNet和VGG-Face模型的组合作为特征提取器,在2020年的Recognizing Family in The Wild比赛中获得了第9名的亲属关系验证和第5名的亲属关系检索。然后,我们使用StyleGAN 2作为另一个编码器进行了进一步的实验,结果没有任何改善。
Contrastive masked auto-encoders based self-supervised hashing for 2D image and 3D point cloud cross-modal retrieval
Wuhan National Laboratory for Optoelectronics, Huazhong University of Science and Technology(华中科技大学武汉光电国家研究中心);School of Computer Science and Technology, Huazhong University of Science and Technology(华中科技大学计算机科学与技术学院);School of Software Engineering, Huazhong University of Science and Technology(华中科技大学软件学院);School of Computer Science and Cyber Engineering, Guangzhou University(广州大学计算机科学与网络工程学院)
Exploring the Capabilities of Large Language Model Encoders for Image-Text Retrieval in Chest X-rays
Interdisciplinary Program in Bioengineering, Seoul National University Graduate School(生物工程跨学科项目,首尔国立大学研究生院);Integrated Major in Innovative Medical Science, Seoul National University Graduate School(创新医学科学整合专业,首尔国立大学研究生院);Department of Radiology, The First Affiliated Hospital, Zhejiang University School of Medicine(浙江大学医学院第一附属医院放射科);Seoul National University College of Medicine(首尔国立大学医学院);Department of Radiology, Seoul National University College of Medicine, Seoul National University Hospital(首尔国立大学医学院放射科,首尔国立大学医院);Institute of Medical and Biological Engineering, Seoul National University Medical Research Center(医学与生物工程研究所,首尔国立大学医学研究所以及);Institute of Radiation Medicine, Seoul National University Medical Research Center(放射医学研究所,首尔国立大学医学研究所以及)
School of Artificial Intelligence and Robotics, Hunan University(湖南大学人工智能与机器人学院);College of Computer Science and Software Engineer, Shenzhen University(深圳大学计算机科学与软件工程学院);Faculty of Science and Technology, University of Macau(澳门大学科学与技术学院)
BRIDGE: Multimodal-to-Text Retrieval via Reinforcement-Learned Query Alignment
High institute for computer & information systems(高等计算机与信息系统学院);Chungbuk National University(忠北大学);Assiut University(艾斯尤特大学);University of Innsbruck(因斯布鲁克大学)