What is the interplay between semantic representations learned by language models (LM) from surface form alone to those learned from more grounded evidence? We study this question for a scenario where part of the input comes from a different modality -- in our case, in a vision-language model (VLM), where a pretrained LM is aligned with a pretrained image encoder. As a case study, we focus on the task of predicting hypernyms of objects represented in images. We do so in a VLM setup where the image encoder and LM are kept frozen, and only the intermediate mappings are learned. We progressively deprive the VLM of explicit evidence for hypernyms, and test whether knowledge of hypernyms is recoverable from the LM. We find that the LMs we study can recover this knowledge and generalize even in the most extreme version of this experiment (when the model receives no evidence of a hypernym during training). Additional experiments suggest that this cross-modal taxonomic generalization persists under counterfactual image-label mappings only when the counterfactual data have high visual similarity within each category. Taken together, these findings suggest that cross-modal generalization in LMs arises as a result of both coherence in the extralinguistic input and knowledge derived from language cues.
Vision-and-Language Training Helps Deploy Taxonomic Knowledge but Does Not Fundamentally Alter It
Boston University(波士顿大学);University of Amsterdam(阿姆斯特丹大学);Umeå University(乌梅大学);Vrije Universiteit Amsterdam(阿姆斯特丹自由大学);TTIC(TTIC研究中心);The University of Texas at Austin(德克萨斯大学奥斯汀分校)
MMICL: Empowering Vision-language Model with Multi-Modal In-Context Learning
Peking University(北京大学);School of Software and Microelectronics, Peking University(北京大学软件与微电子学院);BIGAI(北京智源人工智能研究院);Paul G. Allen School of Computer Science and Engineering, University of Washington(华盛顿大学保罗·G·艾伦计算机科学与工程学院);Beijing Jiaotong University(北京交通大学)
COSMOS: Cross-Modality Self-Distillation for Vision Language Pre-training
Technical University of Munich(慕尼黑工业大学);Munich Center for Machine Learning (MCML)(慕尼黑机器学习中心);Helmholtz Munich(亥姆霍兹慕尼黑中心);Munich Data Science Institute (MDSI)(慕尼黑数据科学研究所)
3D Modality-Aware Pre-training for Vision-Language Model in MRI Multi-organ Abnormality Detection
School of Electronic, Electrical Engineering and Physics, Fujian University of Technology(福建工程学院电子电气工程学院);School of Computer Science and Engineering, Southeast University, China(东南大学计算机科学与工程学院);Department of Medical Imaging, Suzhou Traditional Chinese Medicine Hospital, China(苏州中医医院影像科)
Stream-Omni: Simultaneous Multimodal Interactions with Large Language-Vision-Speech Model
Key Laboratory of Intelligent Information Processing, Institute of Computing Technology, Chinese Academy of Sciences (ICT/CAS)(中国科学院智能信息处理重点实验室);Key Laboratory of AI Safety, Chinese Academy of Sciences(中国科学院人工智能安全重点实验室);University of Chinese Academy of Sciences, Beijing, China(中国科学院大学)
Breaking Modality Heterogeneity in Low-Bit Quantization for Large Vision-Language Models
VCIP, College of Computer Science, Nankai University(南开大学计算机科学学院VCIP);D-ITET, ETH Zürich(苏黎世联邦理工学院D-ITET);OPPO Research Institute(OPPO研究院);Department of Computing, Hong Kong Polytechnic University(香港理工大学计算机系)
Unveiling the Fragility of Vision-Language Models: Multi-Modal Adversarial Synergy via Texture-Constrained Perturbations and Cross-Modal Optimization
School of Software Engineering, Huazhong University of Science and Technology(华中科技大学软件学院);Nanyang Technological University, Singapore(新加坡南洋理工大学);University College London(伦敦大学学院)
Red-teaming the Multimodal Reasoning: Jailbreaking Vision-Language Models via Cross-modal Entanglement Attacks
Institute of Computing Technology, Chinese Academy of Sciences(中国科学院计算技术研究所);University of Chinese Academy of Sciences(中国科学院大学);People’s Public Security University of China(中国人民公安大学)
CoMT: A Novel Benchmark for Chain of Multi-modal Thought on Large Vision-Language Models
School of Computer Science and Engineering, Central South University(中南大学计算机科学与工程学院);Key Laboratory of Data Intelligence and Advanced Computing in Provincial Universities, Soochow University(苏州大学省级高校数据智能与先进计算重点实验室);Research Center for SCIR, Harbin Institute of Technology(哈尔滨工业大学SCIR研究中心);National University of Singapore(新加坡国立大学)
Do Vision-Language Models Truly Perform Vision Reasoning? A Rigorous Study of the Modality Gap
College of Computing and Data Science, Nanyang Technological University, Singapore(南洋理工大学计算机与数据科学学院,新加坡);Alibaba-NTU Global e-Sustainability CorpLab (ANGEL)(阿里巴巴-国立大学全球可持续发展公司实验室(ANGEL));Tongyi Lab, Alibaba Group, China(阿里云实验室,阿里巴巴集团,中国)
CLIBD: Bridging Vision and Genomics for Biodiversity Monitoring at Scale
Simon Fraser University(西蒙弗雷泽大学);Aalborg University(奥胡斯大学);Vector Institute(向量研究所);University of Guelph(圭尔夫大学);Alberta Machine Intelligence Institute (Amii)(阿尔伯塔人工智能研究所)