A method for creating a vision-and-language (V&L) model is to extend a language model through structural modifications and V&L pre-training. Such an extension aims to make a V&L model inherit the capability of natural language understanding (NLU) from the original language model. To see how well this is achieved, we propose to evaluate V&L models using an NLU benchmark (GLUE). We compare five V&L models, including single-stream and dual-stream models, trained with the same pre-training. Dual-stream models, with their higher modality independence achieved by approximately doubling the number of parameters, are expected to preserve the NLU capability better. Our main finding is that the dual-stream scores are not much different than the single-stream scores, contrary to expectation. Further analysis shows that pre-training causes the performance drop in NLU tasks with few exceptions. These results suggest that adopting a single-stream structure and devising the pre-training could be an effective method for improving the maintenance of language knowledge in V&L extensions.
Visual Riddles: a Commonsense and World Knowledge Challenge for Large Vision and Language Models
Ben Gurion University(本古里安大学);Bar-Ilan University(巴伊兰大学);The Hebrew University of Jerusalem(耶路撒冷希伯来大学);Google Research(谷歌研究院);Tel Aviv University(特拉维夫大学)
CoViPAL: Layer-wise Contextualized Visual Token Pruning for Large Vision-Language Models
School of Artificial Intelligence, Wuhan University(武汉大学人工智能学院);School of Computer Science, Wuhan University(武汉大学计算机学院);School of Computer Science, Shanghai Jiao Tong University(上海交通大学计算机学院);Cognitive AI Lab, Shanghai Huawei Technologies, China(上海华为技术有限公司认知人工智能实验室)
VinVL: Revisiting Visual Representations in Vision-Language Models
Microsoft Corporation(微软公司);University of Washington(华盛顿大学)
本文提出了一个详细的研究,改善视觉语言(VL)任务的视觉表示,并开发了一个改进的目标检测模型,以提供以对象为中心的图像表示。与最广泛使用的\emph{bottom-up and top-down}模型\cite{anderson2018bottom}相比,新模型更大,针对VL任务设计得更好,并且在更大的训练语料库上进行了预训练,这些训练语料库结合了多个公共注释的目标检测数据集。因此,它可以生成更丰富的视觉对象和概念集合的表示。虽然以前的VL研究主要集中在改善视觉语言融合模型,并离开目标检测模型的改进不变,我们表明,视觉功能的重要性显着VL模型。在我们的实验中,我们将由新的目标检测模型生成的视觉特征馈送到基于transformer的VL融合模型\oscar \cite{li2020oscar}中,并利用改进的方法\short \来预训练VL模型并在广泛的下游VL任务上对其进行微调。我们的研究结果表明,新的视觉功能显着提高了所有VL任务的性能,在七个公共基准测试中创造了新的最先进的结果。我们将向公众发布新的目标检测模型。
VCM: Vision Concept Modeling Based on Implicit Contrastive Learning with Vision-Language Instruction Fine-Tuning
Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences(中国科学院深圳先进技术研究所);University of Chinese Academy of Sciences(中国科学院大学);National University of Singapore(新加坡国立大学);MoE Key Laboratory of Brain-inspired Intelligent Perception and Cognition, University of Science and Technology of China(中国科学技术大学脑启发智能感知与认知教育部重点实验室)
Look Again, Think Slowly: Enhancing Visual Reflection in Vision-Language Models
Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所);School of Artificial Intelligence, University of Chinese Academy of Sciences(中国科学院大学人工智能学院);Wuhan AI Research(武汉人工智能研究院)
UniVLR: Unifying Text and Vision in Visual Latent Reasoning for Multimodal LLMs
University of Science and Technology of China(中国科学技术大学);Tsinghua University(清华大学);National University of Singapore(新加坡国立大学);Zhongguancun Academy(中关村学院)
Do we Really Need Visual Instructions? Towards Visual Instruction-Free Fine-tuning for Large Vision-Language Models
Gaoling School of Artificial Intelligence, Renmin University of China(中国人民大学高瓴人工智能学院);School of Information, Renmin University of China(中国人民大学信息学院);Alibaba Group(阿里巴巴集团)
Memory-Space Visual Prompting for Efficient Vision-Language Fine-Tuning
School of Intelligence Science and Technology, Peking University(北京大学智能科学与技术学院);National Key Laboratory of General Artificial Intelligence(通用人工智能全国重点实验室)