Due to the amount of videos and related captions uploaded every hour, deep learning-based solutions for cross-modal video retrieval are attracting more and more attention. A typical approach consists in learning a joint text-video embedding space, where the similarity of a video and its associated caption is maximized, whereas a lower similarity is enforced with all the other captions, called negatives. This approach assumes that only the video and caption pairs in the dataset are valid, but different captions - positives - may also describe its visual contents, hence some of them may be wrongly penalized. To address this shortcoming, we propose the Relevance-Aware Negatives and Positives mining (RANP) which, based on the semantics of the negatives, improves their selection while also increasing the similarity of other valid positives. We explore the influence of these techniques on two video-text datasets: EPIC-Kitchens-100 and MSR-VTT. By using the proposed techniques, we achieve considerable improvements in terms of nDCG and mAP, leading to state-of-the-art results, e.g. +5.3% nDCG and +3.0% mAP on EPIC-Kitchens-100. We share code and pretrained models at \url{https://github.com/aranciokov/ranp}.
Semantic Hardness Is Not Visual Hardness: Sign-Aware Hard Negative Mining for Sign Language Retrieval
School of Computing(计算机学院);Graduate School of Culture Technology(文化技术研究生院);Korea Advanced Institute of Science and Technology(韩国科学技术院);ETRI Medical Informatics Laboratory(电子通信研究院医学信息学实验室)
SMc2f: Robust Scenario Mining for Robotic Autonomy from Coarse to Fine
Department of Computer Science at Xi’an University of Technology(西安理工大学计算机科学系);department of Computer Science & Engineering at the University of California, Merced(加州大学默塞德分校计算机科学与工程系)
Relevance-based Margin for Contrastively-trained Video Retrieval Models
Fondazione Bruno Kessler(布鲁诺·凯斯勒基金会);University of Udine(乌迪内大学);Samsung AI Center Cambridge(三星剑桥人工智能中心);University of Barcelona(巴塞罗那大学);Computer Vision Center(计算机视觉中心);Free University of Bozen-Bolzano(博尔扎诺自由大学)
MSAM: Multi-Semantic Adaptive Mining for Cross-Modal Drone Video-Text Retrieval
School of Computer Science and Engineering, Sun Yat-sen University(中山大学计算机科学与工程学院);School of Computer Science and Artificial Intelligence, Wuhan University of Technology(武汉理工大学计算机科学与人工智能学院);School of Artificial Intelligence, Optics and Electronics (iOPEN), Northwestern Polytechnical University(西北工业大学人工智能、光学与电子学院(iOPEN))
Institute of Computing Technology, Chinese Academy of Sciences(中国科学院计算技术研究所);University of Chinese Academy of Sciences(中国科学院大学);Gaoling School of Artificial Intelligence, Renmin University of China(中国人民大学高瓴人工智能学院)
Query-free Clothing Retrieval via Implicit Relevance Feedback
Cooperative Medianet Innovation Center, Shanghai Jiao Tong University(上海交通大学协同媒体创新中心);Institute of Image Communication and Network Engineering, Shanghai Jiao Tong University(上海交通大学图像通信与网络工程研究所)
Attributes Grouping and Mining Hashing for Fine-Grained Image Retrieval
Southeast University(东南大学);Key Laboratory of Measurement and Control of Complex Systems of Engineering, Ministry of Education(教育部复杂工程系统测量与控制重点实验室);Nanjing Enbo Technology Co., Ltd.(南京恩博科技有限公司)