For a vision-language model (VLM) to understand the physical world, such as cause and effect, a first step is to capture the temporal dynamics of the visual world, for example how the physical states of objects evolve over time (e.g. a whole apple into a sliced apple). Our paper aims to investigate if VLMs pre-trained on web-scale data learn to encode object states, which can be extracted with zero-shot text prompts. We curate an object state recognition dataset ChangeIt-Frames, and evaluate nine open-source VLMs, including models trained with contrastive and generative objectives. We observe that while these state-of-the-art vision-language models can reliably perform object recognition, they consistently fail to accurately distinguish the objects' physical states. Through extensive experiments, we identify three areas for improvements for VLMs to better encode object states, namely the quality of object localization, the architecture to bind concepts to objects, and the objective to learn discriminative visual and language encoders on object states. Data and code are released.
Enhancing Perception Capabilities of Multimodal LLMs with Training-Free Fusion
ZIP Lab, Zhejiang University（浙江大学 ZIP 实验室）；South China University of Technology（华南理工大学）；Pazhou Lab（琶洲实验室）；Key Laboratory of Big Data and Intelligent Robot, Ministry of Education（大数据与智能机器人教育部重点实验室）
Securing Vision-Language Models with a Robust Encoder Against Jailbreak and Adversarial Attacks
School of Computing, Southern Illinois University（南伊利诺伊大学计算学院）；Security, Privacy and Intelligence for Edge Devices Laboratory (SPEED Lab)（边缘设备安全、隐私与智能实验室（SPEED Lab））
Investigating Redundancy in Multimodal Large Language Models with Multiple Vision Encoders
Shanghai Artificial Intelligence Laboratory（上海人工智能实验室）；The Hong Kong University of Science and Technology (Guangzhou)（香港科技大学（广州））；Zhejiang University（浙江大学）；Beijing University of Technology（北京工业大学）
CoVisco: Codec-Native Vision Encoder with Native Token Compression for Unified Image-Video Understanding
ERNIE Team, Baidu Inc.（百度公司ERNIE团队）；The Hong Kong University of Science and Technology（香港科技大学）；Institute of Automation, Chinese Academy of Sciences (CASIA)（中国科学院自动化研究所）
The Vision Encoder as a Privacy Boundary: Visual-Token Side Channels in Encoder-Free Vision-Language Models
School of Engineering, Institute of Science Tokyo（东京科学大学工学院）；College of Control Science and Engineering, Zhejiang University（浙江大学控制科学与工程学院）；Department of Electrical and Computer Engineering, National University of Singapore（新加坡国立大学电气与计算机工程系）
Harbin Institute of Technology（哈尔滨工业大学）；Microsoft Research（微软研究院）；Peng Cheng Laboratory（鹏城实验室）
We propose a cross-modal attention distillation framework to train a dual-encoder model for vision-language understanding tasks, such as visual reasoning and visual question answering. Dual-encoder models have a faster inference speed than fusion-encoder models and enable the pre-computation of images and text during inference. However, the shallow interaction module used in dual-encoder models is insufficient to handle complex vision-language understanding tasks. In order to learn deep interactions of images and text, we introduce cross-modal attention distillation, which uses the image-to-te…
Columbia University（哥伦比亚大学）；Microsoft Research（微软研究院）；University of California, Los Angeles（加利福尼亚大学洛杉矶分校）
Cross-modal encoders for vision-language (VL) tasks are often pretrained with carefully curated vision-language datasets. While these datasets reach an order of 10 million samples, the labor cost is prohibitive to scale further. Conversely, unimodal encoders are pretrained with simpler annotations that are less cost-prohibitive, achieving scales of hundreds of millions to billions. As a result, unimodal encoders have achieved state-of-art (SOTA) on many downstream tasks. However, challenges remain when applying to VL tasks. The pretraining data is not optimal for cross-modal architectures and …
University of Trento（特伦托大学）；University of Amsterdam（阿姆斯特丹大学）
The multimodal models used in the emerging field at the intersection of computational linguistics and computer vision implement the bottom-up processing of the `Hub and Spoke' architecture proposed in cognitive science to represent how the brain processes and combines multi-sensory inputs. In particular, the Hub is implemented as a neural network encoder. We investigate the effect on this encoder of various vision-and-language tasks proposed in the literature: visual question answering, visual reference resolution, and visually grounded dialogue. To measure the quality of the representations l…
GeoDANO: Geometric VLM with Domain Agnostic Vision Encoder
Graduate School of Artificial Intelligence, POSTECH（POSTECH人工智能研究生院）；Department of Computer Science and Engineering, POSTECH（POSTECH计算机科学与工程系）；Australian National University（澳大利亚国立大学）
JD AI Research（京东人工智能研究院）；Shanghai Jiao Tong University（上海交通大学）
Vision-language pre-training has been an emerging and fast-developing research topic, which transfers multi-modal knowledge from rich-resource pre-training task to limited-resource downstream tasks. Unlike existing works that predominantly learn a single generic encoder, we present a pre-trainable Universal Encoder-DEcoder Network (Uni-EDEN) to facilitate both vision-language perception (e.g., visual question answering) and generation (e.g., image captioning). Uni-EDEN is a two-stream Transformer based structure, consisting of three modules: object and sentence encoders that separately learns …
Expanding the Boundaries of Vision Prior Knowledge in Multi-modal Large Language Models
University of Chinese Academy of Sciences（中国科学院大学）；Chinese Information Processing Laboratory, Institute of Software, Chinese Academy of Sciences（中国科学院软件研究所信息处理实验室）
Peking University（北京大学）；Microsoft Research Asia（微软亚洲研究院）；Microsoft（微软）
We propose Unicoder-VL, a universal encoder that aims to learn joint representations of vision and language in a pre-training manner. Borrow ideas from cross-lingual pre-trained models, such as XLM and Unicoder, both visual and linguistic contents are fed into a multi-layer Transformer for the cross-modal pre-training, where three pre-trained tasks are employed, including Masked Language Modeling (MLM), Masked Object Classification (MOC) and Visual-linguistic Matching (VLM). The first two tasks learn context-aware representations for input tokens based on linguistic and visual contents jointly…
