Existing vision-language methods typically support two languages at a time at most. In this paper, we present a modular approach which can easily be incorporated into existing vision-language methods in order to support many languages. We accomplish this by learning a single shared Multimodal Universal Language Embedding (MULE) which has been visually-semantically aligned across all languages. Then we learn to relate MULE to visual data as if it were a single language. Our method is not architecture specific, unlike prior work which typically learned separate branches for each language, enabling our approach to easily be adapted to many vision-language methods and tasks. Since MULE learns a single language branch in the multimodal model, we can also scale to support many languages, and languages with fewer annotations can take advantage of the good representation learned from other (more abundant) language data. We demonstrate the effectiveness of MULE on the bidirectional image-sentence retrieval task, supporting up to four languages in a single model. In addition, we show that Machine Translation can be used for data augmentation in multilingual learning, which, combined with MULE, improves mean recall by up to 21.9% on a single-language compared to prior work, with the most significant gains seen on languages with relatively few annotations. Our code is publicly available.
Shanghai Jiao Tong University（上海交通大学）；Harbin Institute of Technology, Shenzhen（哈尔滨工业大学（深圳））；National Institute of Information and Communications Technology（日本国立信息通信研究机构）
Representation learning is the foundation of natural language processing (NLP). This work presents new methods to employ visual information as assistant signals to general NLP tasks. For each sentence, we first retrieve a flexible number of images either from a light topic-image lookup table extracted over the existing sentence-image pairs or a shared cross-modal embedding space that is pre-trained on out-of-shelf text-image pairs. Then, the text and images are encoded by a Transformer encoder and convolutional neural network, respectively. The two sequences of representations are further fuse…
Harbin Institute of Technology（哈尔滨工业大学）；Microsoft Research Asia（微软亚洲研究院）；Microsoft（微软）；Microsoft Research Redmond（微软研究院（雷德蒙德））
We present M3P, a Multitask Multilingual Multimodal Pre-trained model that combines multilingual pre-training and multimodal pre-training into a unified framework via multitask pre-training. Our goal is to learn universal representations that can map objects occurred in different modalities or texts expressed in different languages into a common semantic space. In addition, to explicitly encourage fine-grained alignment between images and non-English languages, we also propose Multimodal Code-switched Training (MCT) to combine monolingual pre-training and multimodal pre-training via a code-swi…
m3P: Towards Multimodal Multilingual Translation with Multimodal Prompt
State Key Lab of Software Development Environment, Beihang University（北京航空航天大学软件开发环境国家重点实验室）；Department of Computer Science, University of British Columbia（不列颠哥伦比亚大学计算机科学系）
Exploring the Necessity of Visual Modality in Multimodal Machine Translation using Authentic Datasets
College of Big Data and Internet, Shenzhen Technology University（深圳技术大学大数据与互联网学院）；College of Application and Technology, Shenzhen University（深圳大学应用技术学院）
Hierarchical Refinement of Universal Multimodal Attacks on Vision-Language Models
School of Electrical Engineering and Computer Science, the University of Queensland（电气工程与计算机科学学院，昆士兰大学）；Department of Computer Science, City University of Hong Kong（计算机科学系，香港城市大学）
UniIR: Training and Benchmarking Universal Multimodal Information Retrievers
University of Waterloo（滑铁卢大学）；Georgia Institute of Technology（佐治亚理工学院）；Hong Kong University of Science and Technology（香港科技大学）；Google DeepMind（谷歌DeepMind）
MegaPairs: Massive Data Synthesis For Universal Multimodal Retrieval
Beijing University of Posts and Telecommunications（北京邮电大学）；Beijing Academy of Artificial Intelligence（北京人工智能研究院）；The Hong Kong Polytechnic University（香港理工大学）；University of Science and Technology of China（中国科学技术大学）；Shanghai Jiaotong University（上海交通大学）
V-Retrver: Evidence-Driven Agentic Reasoning for Universal Multimodal Retrieval
Tsinghua University（清华大学）；University of Central Florida（中央佛罗里达大学）；Fudan University（复旦大学）；The Australian National University（澳大利亚国立大学）；The University of Hong Kong（香港大学）；Pengcheng Laboratory（鹏城实验室）；Central South University（中南大学）
Zhejiang University-University of Illinois at Urbana-Champaign Institute（浙江大学-伊利诺伊大学厄巴纳香槟分校联合学院）；Zhejiang University（浙江大学）
Multimodal sentiment analysis (MSA) is an important way of observing mental activities with the help of data captured from multiple modalities. However, due to the recording or transmission error, some modalities may include incomplete data. Most existing works that address missing modalities usually assume a particular modality is completely missing and seldom consider a mixture of missing across multiple modalities. In this paper, we propose a simple yet effective meta-sampling approach for multimodal sentiment analysis with missing modalities, namely Missing Modality-based Meta Sampling (M3…
mChartQA: A universal benchmark for multimodal Chart Question Answer based on Vision-Language Alignment and Reasoning
Shenyang Institute of Computing Technology, Chinese Academy of Sciences（中国科学院沈阳计算技术研究所）；Beijing Wenge Technology Co., Ltd（北京文格科技有限公司）；University of Chinese Academy of Sciences（中国科学院大学）
Shanghai Jiao Tong University（上海交通大学）；National Institute of Information（国立信息研究所）
Word representation is a fundamental component in neural language understanding models. Recently, pre-trained language models (PrLMs) offer a new performant method of contextualized word representations by leveraging the sequence-level context for modeling. Although the PrLMs generally give more accurate contextualized word representations than non-contextualized models do, they are still subject to a sequence of text contexts without diverse hints for word representation from multimodality. This paper thus proposes a visual representation method to explicitly enhance conventional word embeddi…
Autoregressive Retriever: Improving Query Understanding from Item Feedback for Universal Multimodal Retrieval
School of Computer Science and Technology, Beijing Institute of Technology（北京理工大学计算机科学与技术学院）；Zhongguancun Academy（中关村实验室）；Xiaohongshu（小红书）；Southeast Academy of Information Technology, Beijing Institute of Technology（北京理工大学东南信息科学研究院）；Zhongguancun Laboratory（中关村实验室）
UniRef-UAV: A Multimodal Benchmark for Universal Referring in UAV Imagery
School of Automation, Northwestern Polytechnical University（自动化学院，西北工业大学）；College of Missile Engineering, Rocket Force University of Engineering（导弹工程学院，火箭兵工程大学）；School of Artificial Intelligence, Chongqing University of Posts and Telecommunications（人工智能学院，重庆邮电大学）
School of Traffic and Transportation, Beijing Jiaotong University（交通与运输学院，北京交通大学）；Foundation Modal Research Center, Institute of Automation, Chinese Academy of Sciences（基础模态研究中心，自动化研究所）；Queen Mary School Hainan, Beijing University of Posts and Telecommunications（海南女王学院，北京邮电大学）；Department of Computer Science, NUS School of Computing（计算机科学系，国立大学计算机学院）
multiGradICON: A Foundation Model for Multimodal Medical Image Registration
University of North Carolina at Chapel Hill（北卡罗来纳大学教堂山分校）；University of Salzburg（萨尔茨堡大学）；Université Gustave Eiffel（古斯塔夫·埃菲尔大学）；Brigham and Women’s Hospital（布莱根妇女医院）；ÉTS Montréal（蒙特利尔工程学院）；Boston University（波士顿大学）；Kitware Inc.（Kitware公司）
Latent text representations exhibit geometric regularities, such as the famous analogy: queen is to king what woman is to man. Such structured semantic relations were not demonstrated on image representations. Recent works aiming at bridging this semantic gap embed images and text into a multimodal space, enabling the transfer of text-defined transformations to the image modality. We introduce the SIMAT dataset to evaluate the task of Image Retrieval with Multimodal queries. SIMAT contains 6k images and 18k textual transformation queries that aim at either replacing scene elements or changing …
