Numerous recent works have proposed pretraining generic visio-linguistic representations and then finetuning them for downstream vision and language tasks. While architecture and objective function design choices have received attention, the choice of pretraining datasets has received little attention. In this work, we question some of the default choices made in literature. For instance, we systematically study how varying similarity between the pretraining dataset domain (textual and visual) and the downstream domain affects performance. Surprisingly, we show that automatically generated data in a domain closer to the downstream task (e.g., VQA v2) is a better choice for pretraining than "natural" data but of a slightly different domain (e.g., Conceptual Captions). On the other hand, some seemingly reasonable choices of pretraining datasets were found to be entirely ineffective for some downstream tasks. This suggests that despite the numerous recent efforts, vision & language pretraining does not quite work "out of the box" yet. Overall, as a by-product of our study, we find that simple design choices in pretraining can help us achieve close to state-of-art results on downstream tasks without any architectural changes.
We introduce a vision-language foundation model called VL-BEiT, which is a bidirectional multimodal Transformer learned by generative pretraining. Our minimalist solution conducts masked prediction on both monomodal and multimodal data with a shared Transformer. Specifically, we perform masked vision-language modeling on image-text pairs, masked language modeling on texts, and masked image modeling on images. VL-BEiT is learned from scratch with one unified pretraining task, one shared backbone, and one-stage training. Our method is conceptually simple and empirically effective. Experimental r…
A big convergence of language, vision, and multimodal pretraining is emerging. In this work, we introduce a general-purpose multimodal foundation model BEiT-3, which achieves state-of-the-art transfer performance on both vision and vision-language tasks. Specifically, we advance the big convergence from three aspects: backbone architecture, pretraining task, and model scaling up. We introduce Multiway Transformers for general-purpose modeling, where the modular architecture enables both deep fusion and modality-specific encoding. Based on the shared backbone, we perform masked "language" model…
Georgia Institute of Technology（佐治亚理工学院）；Facebook AI Research（脸书人工智能研究院）；Oregon State University（俄勒冈州立大学）
We present ViLBERT (short for Vision-and-Language BERT), a model for learning task-agnostic joint representations of image content and natural language. We extend the popular BERT architecture to a multi-modal two-stream model, pro-cessing both visual and textual inputs in separate streams that interact through co-attentional transformer layers. We pretrain our model through two proxy tasks on the large, automatically collected Conceptual Captions dataset and then transfer it to multiple established vision-and-language tasks -- visual question answering, visual commonsense reasoning, referring…
Linguistic representations derived from text alone have been criticized for their lack of grounding, i.e., connecting words to their meanings in the physical world. Vision-and-Language (VL) models, trained jointly on text and image or video data, have been offered as a response to such criticisms. However, while VL pretraining has shown success on multimodal tasks such as visual question answering, it is not yet known how the internal linguistic representations themselves compare to their text-only counterparts. This paper compares the semantic representations learned via VL vs. text-only pret…
Nanyang Technological University（南洋理工大学）；Agency for Science, Technology and Research（新加坡科技研究局）
Current vision language pretraining models are dominated by methods using region visual features extracted from object detectors. Given their good performance, the extract-then-process pipeline significantly restricts the inference speed and therefore limits their real-world use cases. However, training vision language models from raw image pixels is difficult, as the raw image pixels give much less prior knowledge than region features. In this paper, we systematically study how to leverage auxiliary visual pretraining tasks to help training end-to-end vision language models. We introduce thre…
Zhejiang University（浙江大学）；University of Electronic Science and Technology of China（电子科技大学）；Alibaba Group（阿里巴巴集团）
In this paper, we propose to investigate the problem of out-of-domain visio-linguistic pretraining, where the pretraining data distribution differs from that of downstream data on which the pretrained model will be fine-tuned. Existing methods for this problem are purely likelihood-based, leading to the spurious correlations and hurt the generalization ability when transferred to out-of-domain downstream tasks. By spurious correlation, we mean that the conditional probability of one token (object or word) given another one can be high (due to the dataset biases) without robust (causal) relatio…
The success of large-scale contrastive vision-language pretraining (CLIP) has benefited both visual recognition and multimodal content understanding. The concise design brings CLIP the advantage in inference efficiency against other vision-language models with heavier cross-attention fusion layers, making it a popular choice for a wide spectrum of downstream tasks. However, CLIP does not explicitly capture the hierarchical nature of high-level and fine-grained semantics conveyed in images and texts, which is arguably critical to vision-language understanding and reasoning. To this end, we equi…
Medical image visual question answering (VQA) is a task to answer clinical questions, given a radiographic image, which is a challenging problem that requires a model to integrate both vision and language information. To solve medical VQA problems with a limited number of training data, pretrain-finetune paradigm is widely used to improve the model generalization. In this paper, we propose a self-supervised method that applies Masked image modeling, Masked language modeling, Image text matching and Image text alignment via contrastive learning (M2I2) for pretraining on medical image caption da…
Vision and Language Pretraining has become the prevalent approach for tackling multimodal downstream tasks. The current trend is to move towards ever larger models and pretraining datasets. This computational headlong rush does not seem reasonable in the long term to move toward sustainable solutions, and de facto excludes academic laboratories with limited resources. In this work, we propose a new framework, dubbed ViCHA, that efficiently exploits the input data to boost the learning by: (a) a new hierarchical cross-modal alignment loss, (b) new self-supervised scheme based on masked image mo…
Learning to Read Where to Look: Disease-Aware Vision-Language Pretraining for 3D CT
Computer Vision Group, University of Freiburg, Germany Adaptive \& Agentic AI (A3) Lab, Aarhus University, Denmark Department of Radiology, Medical Center -- University of Freiburg, Germany Chair of Algorithms；Data Structures, University of Freiburg, Germany ELLIS Institute Finland School of Electrical Engineering, Aalto University, Finland
Carnegie Mellon University（卡内基梅隆大学）；Google Research（谷歌研究院）；University of Washington（华盛顿大学）
With recent progress in joint modeling of visual and textual representations, Vision-Language Pretraining (VLP) has achieved impressive performance on many multimodal downstream tasks. However, the requirement for expensive annotations including clean image captions and regional labels limits the scalability of existing approaches, and complicates the pretraining procedure with the introduction of multiple dataset-specific objectives. In this work, we relax these constraints and present a minimalist pretraining framework, named Simple Visual Language Model (SimVLM). Unlike prior work, SimVLM r…
From Panel to Pixel: Zoom-In Vision-Language Pretraining from Biomedical Scientific Literature
University of Strasbourg, CNRS, INSERM, ICube, UMR7357, Strasbourg, France（斯特拉斯堡大学、法国国家科学研究中心、法国国家医学研究院、ICube、UMR7357、斯特拉斯堡，法国）；Technical University of Munich（慕尼黑技术大学）；Stanford University（斯坦福大学）；DSAI, The Hong Kong Polytechnic University（香港理工大学DSAI实验室）；University of British Columbia（不列颠哥伦比亚大学）；Munich Center for Machine Learning（慕尼黑机器学习中心）；IHU Strasbourg, Strasbourg（斯特拉斯堡IHU医院，斯特拉斯堡）；Vector Institute for AI（人工智能向量研究所）；Yale University（耶鲁大学）
Inria（法国国家信息与自动化研究所）；École normale supérieure（巴黎高等师范学院）；CNRS（法国国家科学研究中心）；PSL Research University（巴黎文理研究大学）；IIIT Hyderabad（海得拉巴国际信息技术学院）
Vision-and-language navigation (VLN) aims to enable embodied agents to navigate in realistic environments using natural language instructions. Given the scarcity of domain-specific training data and the high diversity of image and language inputs, the generalization of VLN agents to unseen environments remains challenging. Recent methods explore pretraining to improve generalization, however, the use of generic image-caption datasets or existing small-scale VLN environments is suboptimal and results in limited improvements. In this work, we introduce BnB, a large-scale and diverse in-domain VL…
Northeastern University（东北大学）；NEC Labs America（NEC美国实验室）；UC San Diego（加州大学圣地亚哥分校）
Self-supervised vision-language pretraining from pure images and text with a contrastive loss is effective, but ignores fine-grained alignment due to a dual-stream architecture that aligns image and text representations only on a global level. Earlier, supervised, non-contrastive methods were capable of finer-grained alignment, but required dense annotations that were not scalable. We propose a single stream architecture that aligns images and language at multiple levels: global, fine-grained patch-token, and conceptual/semantic, using two novel tasks: symmetric cross-modality reconstruction (…
CL4D: Contrastive Language-4D Pretraining for Vision-Language Reasoning in Dynamic Scenes
University of Moratuwa（莫拉图瓦大学）；Singapore-MIT Alliance for Research & Technology (SMART) Centre（新加坡-麻省理工研究与技术联盟（SMART）中心）；Agency for Science, Technology and Research (A*STAR)（新加坡科学、技术与研究局（A*STAR））
Location-Aware Pretraining for Medical Difference Visual Question Answering
Department of Electrical and Computer Engineering, Carnegie Mellon University, Kigali, Rwanda（电气与计算机工程系，卡内基梅隆大学，刚果（金）基利齐）；University of Melbourne, Melbourne, Australia（墨尔本大学，墨尔本，澳大利亚）
Large-scale vision-language pre-training has achieved promising results on downstream tasks. Existing methods highly rely on the assumption that the image-text pairs crawled from the Internet are in perfect one-to-one correspondence. However, in real scenarios, this assumption can be difficult to hold: the text description, obtained by crawling the affiliated metadata of the image, often suffers from the semantic mismatch and the mutual compatibility. To address these issues, we introduce PyramidCLIP, which constructs an input pyramid with different semantic levels for each modality, and align…
Automatic text image recognition is a prevalent application in computer vision field. One efficient way is use Convolutional Recurrent Neural Network(CRNN) to accomplish task in an end-to-end(End2End) fashion. However, CRNN notoriously fails to detect multi-row images and excel-like images. In this paper, we present one alternative to first recognize single-row images, then extend the same architecture to recognize multi-row images with proposed multiple methods. To recognize excel-like images containing box lines, we propose Line-Deep Denoising Convolutional AutoEncoder(Line-DDeCAE) to recove…
Universal Pose Pretraining for Generalizable Vision-Language-Action Policies
Tencent Robotics X（腾讯机器人X）；Futian Laboratory（福田实验室）；The Hong Kong University of Science and Technology（香港科学与技术大学）；Fudan University（复旦大学）；Shanghai Innovation Institute（上海创新研究院）
NepAl Applied Mathematics and Informatics Institute for research (NAAMII)（尼泊尔应用数学与信息学研究所（NAAMII））；Rochester Institute of Technology（罗切斯特理工学院）；University of Aberdeen（阿伯丁大学）
Aalto University（阿尔托大学）；Tampere University of Technology（坦佩雷理工大学）
This paper digs deeper into factors that influence egocentric gaze. Instead of training deep models for this purpose in a blind manner, we propose to inspect factors that contribute to gaze guidance during daily tasks. Bottom-up saliency and optical flow are assessed versus strong spatial prior baselines. Task-specific cues such as vanishing point, manipulation point, and hand regions are analyzed as representatives of top-down information. We also look into the contribution of these factors by investigating a simple recurrent neural model for ego-centric gaze prediction. First, deep features …
MedKCO: Medical Vision-Language Pretraining via Knowledge-Driven Cognitive Orchestration
School of Computer Science and Engineering, Southeast University, China（东南大学计算机科学与工程学院）；Key Laboratory of New Generation Artificial Intelligence Technology and Its Interdisciplinary Applications, Ministry of Education, China（教育部新一代人工智能技术及其交叉应用重点实验室）；School of Computer Science and Engineering, Nanjing University of Science and Technology, China（南京理工大学计算机科学与工程学院）
Lost in Distortion: Uncovering the Domain Gap Between Computer Vision and Brain Imaging -- A Study on Pretraining for Age Prediction
Center for Translational Research in Neuroimaging and Data Science（转化神经影像与数据科学研究中心）；College of Artificial Intelligence, Chengdu University of Information Technology（成都信息科技大学人工智能学院）；Department of Applied Mathematics and Statistics（应用数学与统计学系）；McWilliams School of Biomedical Informatics（麦卡威廉生物医学信息学学院）；Department of Applied Mathematics and Theoretical Physics（应用数学与理论物理系）；School of Biomedical Engineering and Imaging Sciences（生物医学工程与影像科学学院）
VALOR: Vision-Audio-Language Omni-Perception Pretraining Model and Dataset
School of Artificial Intelligence, University of Chinese Academy of Sciences（中国科学院大学人工智能学院）；Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；Nanjing University of Science and Technology（南京理工大学）
MM-Retinal V2: Transfer an Elite Knowledge Spark into Fundus Vision-Language Pretraining
School of Computer Science and Engineering, Southeast University（东南大学计算机科学与工程学院）；Department of Ophthalmology, The First Affiliated Hospital of Nanjing Medical University（南京医科大学第一附属医院眼科部）；School of Computer Science and Engineering, Nanjing University of Science and Technology（南京理工大学计算机科学与工程学院）；Institute of High-Performance Computing, Agency for Science, Technology and Research（科技研究局高性能计算研究所）
Vision-Language Pretraining (VLP) and Foundation models have been the go-to recipe for achieving SoTA performance on general benchmarks. However, leveraging these powerful techniques for more complex vision-language tasks, such as cooking applications, with more structured input data, is still little investigated. In this work, we propose to leverage these techniques for structured-text based computational cuisine tasks. Our strategy, dubbed VLPCook, first transforms existing image-text pairs to image and structured-text pairs. This allows to pretrain our VLPCook model using VLP objectives ada…
Recent trends in self-supervised representation learning have focused on removing inductive biases from training pipelines. However, inductive biases can be useful in settings when limited data are available or provide additional insight into the underlying data distribution. We present spatial prior attention (SPAN), a framework that takes advantage of consistent spatial and semantic structure in unlabeled image datasets to guide Vision Transformer attention. SPAN operates by regularizing attention masks from separate transformer heads to follow various priors over semantic regions. These pri…
