Does language help make sense of the visual world? How important is it to actually see the world rather than having it described with words? These basic questions about the nature of intelligence have been difficult to answer because we only had one example of an intelligent system -- humans -- and limited access to cases that isolated language or vision. However, the development of sophisticated Vision-Language Models (VLMs) by artificial intelligence researchers offers us new opportunities to explore the contributions that language and vision make to learning about the world. We ablate components from the cognitive architecture of these models to identify their contributions to learning new tasks from limited data. We find that a language model leveraging all components recovers a majority of a VLM's performance, despite its lack of visual input, and that language seems to allow this by providing access to prior knowledge and reasoning.
Analyzing Fine-Grained Alignment and Enhancing Vision Understanding in Multimodal Language Models
Department of Computer Science and Engineering, The Ohio State University(计算机科学与工程系,俄亥俄州立大学);Translational Data Analytics Institute, The Ohio State University(转化数据分析研究所,俄亥俄州立大学);Department of Biomedical Informatics, The Ohio State University(生物医学信息学系,俄亥俄州立大学)
Analyzing Diffusion and Autoregressive Vision Language Models in Multimodal Embedding Space
Nanyang Technological University(南洋理工大学);Yale University(耶鲁大学);NYU Shanghai(纽约大学上海分校);Alibaba-NTU Singapore Joint Research Institute(阿里-国立新加坡大学联合研究机构);University of the Chinese Academy of Sciences(中国科学院大学);Center for Data Science(数据科学中心);New York University(纽约大学)
Seeing Sarcasm Through Different Eyes: Analyzing Multimodal Sarcasm Perception in Large Vision-Language Models
Anhui Polytechnic University (AHPU)(安徽工程大学);Shanghai University (SHU)(上海大学);Shanghai Jiao Tong University (SJTU)(上海交通大学);Sichuan University (SCU)(四川大学)
ERVQA: A Dataset to Benchmark the Readiness of Large Vision Language Models in Hospital Environments
Indian Institute of Technology, Kharagpur(印度理工学院卡拉格普尔分校);All India Institute of Medical Sciences, Kalyani(卡里亚尼全印度医学科学研究所);All India Institute of Medical Sciences, Deoghar(迪奥加尔全印度医学科学研究所);Dr. B.C. Roy Multi-Speciality Medical Research Centre, IIT Kharagpur(印度理工学院卡拉格普尔分校Dr. B.C. Roy多专科医学研究中心)
Boosting Medical Vision-Language Pretraining via Momentum Self-Distillation under Limited Computing Resources
Faculty of Information Technology, University of Science, Ho Chi Minh City, Vietnam(信息科技学院,科学大学,胡志明市,越南);Vietnam National University, Ho Chi Minh City, Vietnam(越南国家大学,胡志明市,越南)
CapArena: Benchmarking and Analyzing Detailed Image Captioning in the LLM Era
National Key Laboratory for Novel Software Technology, Nanjing University(南京大学计算机软件新技术国家重点实验室);The University of Hong Kong(香港大学);Shanghai Artificial Intelligence Laboratory(上海人工智能实验室)
The Geometry of Representational Failures in Vision Language Models
Dipartimento di Fisica, Università di Torino(都灵大学物理系);Princeton Neuroscience Institute and AI Lab, Princeton University(普林斯顿大学神经科学研究所和AI实验室);Intesa Sanpaolo AI Research(Intesa Sanpaolo AI研究中心);Dipartimento di Scienze Matematiche, Politecnico di Torino(都灵理工学院数学科学系);Network Science Institute, Northeastern University London, UK(伦敦大学东北方大学网络科学研究所)
Image-Based Geolocation Using Large Vision-Language Models
Nanyang Technological University(南洋理工大学);University of New South Wales(新南威尔士大学);Institute of Information Engineering, Chinese Academy of Sciences(中国科学院信息工程研究所)
Indian Institute of Technology Bombay(印度理工学院孟买分校);Department of Electrical Engineering(电气工程系);Mumbai, India(印度孟买)
视觉Transformer(ViTs)在图像处理中的实际应用有限,尽管它们在某些基准上具有最先进的精度。其使用有限的原因包括与卷积神经网络(CNN)相比,它们需要更大的训练数据集和更多的计算资源,这是由于它们的自我注意力机制的二次复杂性。我们提出了一个线性注意力-卷积混合架构-卷积X-formers for Vision(CXV)-来克服这些限制。我们将二次注意力替换为线性注意力机制,例如Performer,Nyströmmormer和Linear Transformer,以减少其GPU使用量。图像数据的归纳先验由卷积子层提供,从而消除了对ViT使用的类标记和位置嵌入的需要。我们还提出了一种新的训练方法,在不同的训练阶段使用两种不同的优化器,并表明它可以提高不同架构中的前1名图像分类精度。在数据和GPU资源(内核、RAM、电源)有限的情况下,CXV优于其他架构、令牌混合器(例如ConvMixer、FNet和MLP Mixer)、Transformer模型(例如ViT、CCT、CvT和混合Xformer)以及用于图像分类的ResNets。
Semi-Supervised Masked Autoencoders: Unlocking Vision Transformer Potential with Limited Data
Department of Electrical and Computer Engineering, Rowan University, Glassboro, NJ, USA(电子与计算机工程系,罗文大学,Glassboro, NJ, USA);Department of Computer Science, Stevens Institute of Technology, Hoboken, NJ, USA(计算机科学系,史蒂文斯理工学院,Hoboken, NJ, USA)
LLM-Guided Diagnostic Evidence Alignment for Medical Vision-Language Pretraining under Limited Pairing
Institute of Intelligent Information Processing, Shanxi University(山西大学智能信息处理研究所);Alliance Manchester Business School, The University of Manchester(曼彻斯特大学阿利安斯商学院);Department of Computer Science and Engineering, The Hong Kong University of Science and Technology(香港科技大学计算机科学与工程系)
3D Adapted Random Forest Vision (3DARFV) for Untangling Heterogeneous-Fabric Exceeding Deep Learning Semantic Segmentation Efficiency at the Utmost Accuracy