Vision-Language Pre-training (VLP) has achieved impressive performance on various cross-modal downstream tasks. However, most existing methods can only learn from aligned image-caption data and rely heavily on expensive regional features, which greatly limits their scalability and performance. In this paper, we propose an end-to-end unified-modal pre-training framework, namely UNIMO-2, for joint learning on both aligned image-caption data and unaligned image-only and text-only corpus. We build a unified Transformer model to jointly learn visual representations, textual representations and semantic alignment between images and texts. In particular, we propose to conduct grounded learning on both images and texts via a sharing grounded space, which helps bridge unaligned images and texts, and align the visual and textual semantic spaces on different types of corpora. The experiments show that our grounded learning method can improve textual and visual semantic alignment for improving performance on various cross-modal tasks. Moreover, benefiting from effective joint modeling of different types of corpora, our model also achieves impressive performance on single-modal visual and textual tasks. Our code and models are public at the UNIMO project page https://unimo-ptm.github.io/.
Mila（米拉研究所）；Université de Montréal（蒙特利尔大学）；ServiceNow Research（ServiceNow研究院）；DeepMind（深度思维）；Samsung - SAIT AI Lab（三星-三星先进技术研究院人工智能实验室）
Large pre-trained models have proved to be remarkable zero- and (prompt-based) few-shot learners in unimodal vision and language tasks. We propose MAPL, a simple and parameter-efficient method that reuses frozen pre-trained unimodal models and leverages their strong generalization capabilities in multimodal vision-language (VL) settings. MAPL learns a lightweight mapping between the representation spaces of unimodal models using aligned image-text data, and can generalize to unseen VL tasks from just a few in-context examples. The small number of trainable parameters makes MAPL effective at lo…
Columbia University（哥伦比亚大学）；Microsoft Research（微软研究院）；University of California, Los Angeles（加利福尼亚大学洛杉矶分校）
Cross-modal encoders for vision-language (VL) tasks are often pretrained with carefully curated vision-language datasets. While these datasets reach an order of 10 million samples, the labor cost is prohibitive to scale further. Conversely, unimodal encoders are pretrained with simpler annotations that are less cost-prohibitive, achieving scales of hundreds of millions to billions. As a result, unimodal encoders have achieved state-of-art (SOTA) on many downstream tasks. However, challenges remain when applying to VL tasks. The pretraining data is not optimal for cross-modal architectures and …
Using Multimodal Deep Neural Networks to Disentangle Language from Visual Aesthetics
Harvard University（哈佛大学）；Harvard School of Public Health（哈佛公共卫生学院）；Massachusetts General Hospital（麻省总医院）；Massachusetts Institute of Technology（麻省理工学院）；Johns Hopkins University（约翰斯·霍普金斯大学）
School of Information, University of Arizona（亚利桑那大学信息学院）
The recently proposed ALFRED challenge task aims for a virtual robotic agent to complete complex multi-step everyday tasks in a virtual home environment from high-level natural language directives, such as "put a hot piece of bread on a plate". Currently, the best-performing models are able to complete less than 5% of these tasks successfully. In this work we focus on modeling the translation problem of converting natural language directives into detailed multi-step sequences of actions that accomplish those goals in the virtual environment. We empirically demonstrate that it is possible to ge…
Aix Marseille Univ（艾克斯-马赛大学）；Université de Toulon（土伦大学）；CNRS（法国国家科学研究中心）
Recent advances in vision-and-language modeling have seen the development of Transformer architectures that achieve remarkable performance on multimodal reasoning tasks. Yet, the exact capabilities of these black-box models are still poorly understood. While much of previous work has focused on studying their ability to learn meaning at the word-level, their ability to track syntactic dependencies between words has received less attention. We take a first step in closing this gap by creating a new multimodal task targeted at evaluating understanding of predicate-noun dependencies in a controll…
Australian National University（澳大利亚国立大学）；University of Adelaide（阿德莱德大学）；Queensland University of Technology（昆士兰科技大学）；Macquarie University（麦考瑞大学）
A robot that can carry out a natural-language instruction has been a dream since before the Jetsons cartoon series imagined a life of leisure mediated by a fleet of attentive robot helpers. It is a dream that remains stubbornly distant. However, recent advances in vision and language methods have made incredible progress in closely related areas. This is significant because a robot interpreting a natural-language navigation instruction on the basis of what it sees is carrying out a vision and language process that is similar to Visual Question Answering. Both tasks can be interpreted as visual…
University of California, Los Angeles（加利福尼亚大学洛杉矶分校）
Cognitive grammar suggests that the acquisition of language grammar is grounded within visual structures. While grammar is an essential representation of natural language, it also exists ubiquitously in vision to represent the hierarchical part-whole structure. In this work, we study grounded grammar induction of vision and language in a joint learning framework. Specifically, we present VLGrammar, a method that uses compound probabilistic context-free grammars (compound PCFGs) to induce the language grammar and the image grammar simultaneously. We propose a novel contrastive learning framewor…
Vision-and-language (V-L) tasks require the system to understand both vision content and natural language, thus learning fine-grained joint representations of vision and language (a.k.a. V-L representations) is of paramount importance. Recently, various pre-trained V-L models are proposed to learn V-L representations and achieve improved results in many tasks. However, the mainstream models process both vision and language inputs with the same set of attention matrices. As a result, the generated V-L representations are entangled in one common latent space. To tackle this problem, we propose D…
We propose UniTAB that Unifies Text And Box outputs for grounded vision-language (VL) modeling. Grounded VL tasks such as grounded captioning require the model to generate a text description and align predicted words with object regions. To achieve this, models must generate desired text and box outputs together, and meanwhile indicate the alignments between words and boxes. In contrast to existing solutions that use multiple separate modules for different outputs, UniTAB represents both text and box outputs with a shared token sequence, and introduces a special <obj> token to naturally indica…
Integrating Unimodal and Vision-Language Representations in Latent Space for Multi-Label Chest X-Ray Classification
Ho Chi Minh City University of Technology（胡志明市理工大学）；Vietnam National University Ho Chi Minh City（越南国立大学胡志明市分校）；AK Technologies Company Limited（AK科技有限公司）
We propose a model to learn visually grounded word embeddings (vis-w2v) to capture visual notions of semantic relatedness. While word embeddings trained using text have been extremely successful, they cannot uncover notions of semantic relatedness implicit in our visual world. For instance, although "eats" and "stares at" seem unrelated in text, they share semantics visually. When people are eating something, they also tend to stare at the food. Grounding diverse relations like "eats" and "stares at" into vision remains challenging, despite recent progress in vision. We note that the visual gr…
Which Way Does Time Flow? A Psychophysics-Grounded Evaluation for Vision-Language Models
Kyoto University（京都大学）；Center for Information and Neural Networks（信息与神经网络中心）；National Institute of Information and Communications Technology（国立信息通信技术研究所）；The University of Osaka（大阪大学）
Scalable Training of Spatially Grounded 2D Vision-Language Models for Radiology
Computer Vision Group, University of Freiburg, Germany（德国弗莱堡大学计算机视觉组）；Department of Radiology, Medical Center -- University of Freiburg, Germany（德国弗莱堡大学医学中心放射科）；CRIION-AI Lab, Freiburg, Germany（德国弗莱堡CRIION-AI实验室）
University of Southern California（南加州大学）；USC Information Sciences Institute（南加州大学信息科学研究所）
Current state-of-the-art vision-and-language models are evaluated on tasks either individually or in a multi-task setting, overlooking the challenges of continually learning (CL) tasks as they arrive. Existing CL benchmarks have facilitated research on task adaptation and mitigating "catastrophic forgetting", but are limited to vision-only and language-only tasks. We present CLiMB, a benchmark to study the challenge of learning multimodal tasks in a CL setting, and to systematically evaluate how upstream continual learning can rapidly generalize to new multimodal and unimodal tasks. CLiMB incl…
From 3D Pose to Prose: Biomechanics-Grounded Vision--Language Coaching
Department of Computer Science, Drexel University（德雷塞尔大学计算机科学系）；Department of Computer Science and Engineering, Michigan State University（密歇根州立大学计算机科学与工程系）
Grounded Entity-Landmark Adaptive Pre-training for Vision-and-Language Navigation
Defense Innovation Institute, Chinese Academy of Military Science（中国军事科学院国防科技创新研究院）；Tianjin Artificial Intelligence Innovation Center（天津人工智能创新中心）；Harbin Institute of Technology (Shenzhen)（哈尔滨工业大学（深圳））
MetaStructAtlas: A Grounded 3D Vision-Language Dataset and Benchmark for Functional and Structural Reasoning in Whole-Body PET/CT
Shanghai Academy of Artificial Intelligence for Science（上海科学人工智能研究院）；Artificial Intelligence Innovation and Incubation Institute, Fudan University（复旦大学人工智能创新与孵化研究院）；Huashan Hospital（华山医院）；Human Phenome Institute, Fudan University（复旦大学人类表型组研究院）；Shanghai Universal Medical Imaging Diagnostic Center（上海全景医学影像诊断中心）
Grounded Knowledge-Enhanced Medical Vision-Language Pre-training for Chest X-Ray
The Chinese University of Hong Kong（香港中文大学）；CU Lab for AI in Radiology (CLAIR), The Chinese University of Hong Kong（香港中文大学放射学人工智能实验室（CLAIR））；Shanghai Jiao Tong University（上海交通大学）；Shanghai AI Laboratory（上海人工智能实验室）；China Unicom Global Limited（中国联通国际有限公司）
