Previous vision-language pre-training models mainly construct multi-modal inputs with tokens and objects (pixels) followed by performing cross-modality interaction between them. We argue that the input of only tokens and object features limits high-level semantic alignment like phrase-to-region grounding. Meanwhile, multi-level alignments are inherently consistent and able to facilitate the representation learning synergistically. Therefore, in this paper, we propose to learn Multi-level semantic alignment for Vision-language Pre-TRaining (MVPTR). In MVPTR, we follow the nested structure of both modalities to introduce concepts as high-level semantics. To ease the learning from multi-modal multi-level inputs, our framework is split into two stages, the first stage focuses on intra-modality multi-level representation learning, the second enforces interactions across modalities via both coarse-grained and fine-grained semantic alignment tasks. In addition to the commonly used image-text matching and masked language model tasks, we introduce a masked concept recovering task in the first stage to enhance the concept representation learning, and two more tasks in the second stage to explicitly encourage multi-level alignments across modalities. Our code is available at https://github.com/Junction4Nako/mvp_pytorch.
Aligning image and text encoders from scratch using contrastive learning requires large amounts of paired image-text data. We alleviate this need by aligning individually pre-trained language and vision representation models using a much smaller amount of paired data, augmented with a curriculum learning algorithm to learn fine-grained vision-language alignments. TOnICS (Training with Ontology-Informed Contrastive Sampling) initially samples minibatches whose image-text pairs contain a wide variety of objects to learn object-level alignment, and progressively samples minibatches where all imag…
LG Electronics（LG电子）；Seoul National University（首尔国立大学）
Human infants learn the names of objects and develop their own conceptual systems without explicit supervision. In this study, we propose methods for learning aligned vision-language conceptual systems inspired by infants' word learning mechanisms. The proposed model learns the associations of visual objects and words online and gradually constructs cross-modal relational graph networks. Additionally, we also propose an aligned cross-modal representation learning method that learns semantic representations of visual objects and words in a self-supervised manner based on the cross-modal relatio…
Self-supervised vision-langage alignment of deep learning representations for bone X-rays analysis
Information and Communication Technologies, Electronics and Applied Mathematics (ICTEAM)（信息、通信技术、电子与应用数学研究所（ICTEAM））；UCLouvain（天主教鲁汶大学）；Service de chirurgie orthopédique et traumatologie, cliniques universitaires Saint-Luc（圣卢克大学医院骨科与创伤外科）；Institut de Recherche Expérimentale et Clinique (IREC)（实验与临床研究所（IREC））
Efficient Medical Vision-Language Alignment Through Adapting Masked Vision Models
School of Informatics, Xiamen University（厦门大学信息学院）；Center for Smart Health, School of Nursing, The Hong Kong Polytechnic University（香港理工大学护理学院智能健康中心）；Department of Biomedical Informatics, Harvard Medical School（哈佛医学院生物医学信息学系）；Department of Radiology, Zhongshan Hospital (Xiamen), Fudan University（复旦大学中山医院放射科）；National Institute for Data Science in Health and Medicine（医学与数据科学国家研究院）；Department of Computer Science, School of Informatics, Xiamen University（厦门大学信息学院计算机科学系）
Vision-and-Language (V+L) pre-training models have achieved tremendous success in recent years on various multi-modal benchmarks. However, the majority of existing models require pre-training on a large set of parallel image-text data, which is costly to collect, compared to image-only or text-only data. In this paper, we explore unsupervised Vision-and-Language pre-training (UVLP) to learn the cross-modal representation from non-parallel image and text datasets. We found two key factors that lead to good unsupervised V+L pre-training without parallel data: (i) joint image-and-text input (ii) …
Analyzing Fine-Grained Alignment and Enhancing Vision Understanding in Multimodal Language Models
Department of Computer Science and Engineering, The Ohio State University（计算机科学与工程系，俄亥俄州立大学）；Translational Data Analytics Institute, The Ohio State University（转化数据分析研究所，俄亥俄州立大学）；Department of Biomedical Informatics, The Ohio State University（生物医学信息学系，俄亥俄州立大学）
Adaptive Cortically Constrained EEG-Vision Alignment for Zero-Shot Brain-to-Image Retrieval
Chongqing University of Posts and Telecommunications（重庆邮电大学）；Shanghai Jiaotong University（上海交通大学）；Chongqing Normal University（重庆师范大学）；South China University of Technology（华南理工大学）
Do Vision Transformers See Like Humans? Evaluating their Perceptual Alignment
Image Processing Lab, Universidad de Valencia（瓦伦西亚大学图像处理实验室）；Center for Biomaterials and Tissue Engineering Universitat Politecnica de Valencia（多尔多涅理工大学生物材料与组织工程中心）
Vision-R1: Evolving Human-Free Alignment in Large Vision-Language Models via Vision-Guided Reinforcement Learning
Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；University of Chinese Academy of Sciences（中国科学院大学）；Peng Cheng Laboratory（鹏城实验室）；Wuhan AI Research（武汉人工智能研究院）
Decomposed Vision-Language Alignment for Fine-Grained Open-Vocabulary Segmentation
Aerospace Information Research Institute, Chinese Academy of Sciences（中国科学院航空航天信息研究所）；University of Chinese Academy of Sciences（中国科学院大学）；Zhejiang University（浙江大学）
DiscoVLA: Discrepancy Reduction in Vision, Language, and Alignment for Parameter-Efficient Video-Text Retrieval
School of Software（软件学院）；BNRist；Department of Automation, Tsinghua University（自动化系，清华大学）；Hangzhou Zhuoxi Institute of Brain and Intelligence（杭州卓溪脑科学与智能研究院）；GRG Banking Equipment Co., Ltd.（GRG银行设备有限公司）；South China University of Technology（华南理工大学）
ProCLIP: Progressive Vision-Language Alignment via LLM-based Embedder
Shanghai Jiao Tong University（上海交通大学）；Beijing Institute of Technology（北京理工大学）；DeepGlint（深图科技）；Beijing University of Technology（北京工业大学）；Tsinghua University（清华大学）
State-of-the-art vision and vision-and-language models rely on large-scale visio-linguistic pretraining for obtaining good performance on a variety of downstream tasks. Generally, such models are often either cross-modal (contrastive) or multi-modal (with earlier fusion) but not both; and they often only target specific modalities or tasks. A promising direction would be to use a single holistic universal model, as a "foundation", that targets all modalities at once -- a true vision and language foundation model should be good at vision tasks, language tasks, and cross- and multi-modal vision …
VLA-Mark: A cross modal watermark for large vision-language alignment model
The Hong Kong University of Science and Technology (Guangzhou)（香港科技大学（广州））；The Hong Kong University of Science and Technology（香港科技大学）；University of Toronto（多伦多大学）；Ant Group, Alibaba（蚂蚁集团，阿里巴巴）；New York University Shanghai（纽约大学上海分校）
GTA-Net: Cooperative Game Theory for Vision-Language Alignment in Chest X-Ray Report Generation
Department of Computer Science, Rhineland-Palatinate Technical University of Kaiserslautern-Landau（莱茵兰-普法尔茨凯泽斯劳滕-兰道工业大学计算机科学系）；German Research Center for Artificial Intelligence (DFKI)（德国人工智能研究中心）；IntelligentX GmbH
GeoAlignCLIP: Enhancing Fine-Grained Vision-Language Alignment in Remote Sensing via Multi-Granular Consistency Learning
College of Computer Science and Technology, Jilin University, Changchun 130012, China（吉林大学计算机科学与技术学院，长春130012，中国）；Key Laboratory of Symbolic Computation and Knowledge Engineering of Ministry of Education Jilin University（教育部符号计算与知识工程重点实验室，吉林大学）
Multi-task Paired Masking with Alignment Modeling for Medical Vision-Language Pre-training
Hangzhou Dianzi University（杭州电子科技大学）；Zhejiang University（浙江大学）；Sir Run Run Shaw Hospital（邵逸夫医院）；Lenovo Research（联想研究院）；University of Chinese Academy of Sciences（中国科学院大学）；Zhejiang Normal University（浙江师范大学）
TinyAlign: Boosting Lightweight Vision-Language Models by Mitigating Modal Alignment Bottlenecks
Beijing Advanced Innovation Center for Future Blockchain and Privacy Computing（北京未来区块链与隐私计算先进创新中心）；Beihang University（北京航空航天大学）；Hangzhou International Innovation Institute（杭州国际创新研究院）；Xreal；Renmin University（中国人民大学）；Peking University（北京大学）；Beijing Academy of Blockchain and Edge Computing (BABEC)（北京区块链与边缘计算研究院）
