Large Language Models (LLMs) have so far impressed the world, with unprecedented capabilities that emerge in models at large scales. On the vision side, transformer models (i.e., ViT) are following the same trend, achieving the best performance on challenging benchmarks. With the abundance of such unimodal models, a natural question arises; do we need also to follow this trend to tackle multimodal tasks? In this work, we propose to rather direct effort to efficient adaptations of existing models, and propose to augment Language Models with perception. Existing approaches for adapting pretrained models for vision-language tasks still rely on several key components that hinder their efficiency. In particular, they still train a large number of parameters, rely on large multimodal pretraining, use encoders (e.g., CLIP) trained on huge image-text datasets, and add significant inference overhead. In addition, most of these approaches have focused on Zero-Shot and In Context Learning, with little to no effort on direct finetuning. We investigate the minimal computational effort needed to adapt unimodal models for multimodal tasks and propose a new challenging setup, alongside different approaches, that efficiently adapts unimodal pretrained models. We show that by freezing more than 99% of total parameters, training only one linear projection layer, and prepending only one trainable token, our approach (dubbed eP-ALM) significantly outperforms other baselines on VQA and Captioning across Image, Video, and Audio modalities, following the proposed setup. The code is available here: https://github.com/mshukor/eP-ALM.
Perceptual Structure in the Absence of Grounding for LLMs: The Impact of Abstractedness and Subjectivity in Color Language
Rakuten Institute of Technology(乐天技术研究所);Rakuten Group, Inc(乐天集团);National Institute of Advanced Industrial Science and Technology(日本产业技术综合研究所);Graduate School of Engineering, The University of Tokyo(东京大学大学院工学系研究科)
Do Vision Transformers See Like Humans? Evaluating their Perceptual Alignment
Image Processing Lab, Universidad de Valencia(瓦伦西亚大学图像处理实验室);Center for Biomaterials and Tissue Engineering Universitat Politecnica de Valencia(多尔多涅理工大学生物材料与组织工程中心)
Reinforcing Multimodal Reasoning Against Visual Degradation
Tencent Hunyuan(腾讯文言);University of Maryland, College Park(马里兰大学 College Park 分校);University of Virginia(弗吉尼亚大学);University of North Carolina, Chapel Hill(北卡罗来纳大学 Chapel Hill 分校)
A Comprehensive Survey on Video Saliency Detection with Auditory Information: the Audio-visual Consistency Perceptual is the Key!
China University of Petroleum (East China)(中国石油大学(华东));Shandong University of Finance and Economics(山东财经大学);Beijing Information Science and Technology University(北京信息科技大学)