Recent advancements have introduced multiple vision-language models (VLMs) demonstrating impressive commonsense reasoning across various domains. Despite their individual capabilities, the potential of synergizing these complementary VLMs remains underexplored. The Cola Framework addresses this by showcasing how a large language model (LLM) can efficiently coordinate multiple VLMs through natural language communication, leveraging their distinct strengths. We have verified this claim on the challenging A-OKVQA dataset, confirming the effectiveness of such coordination. Building on this, our study investigates whether the same methodology can be applied to surveillance videos for action recognition. Specifically, we explore if leveraging the combined knowledge base of VLMs and LLM can effectively deduce actions from a video when presented with only a few selectively important frames and minimal temporal information. Our experiments demonstrate that LLM, when coordinating different VLMs, can successfully recognize patterns and deduce actions in various scenarios despite the weak temporal signals. However, our findings suggest that to enhance this approach as a viable alternative solution, integrating a stronger temporal signal and exposing the models to slightly more frames would be beneficial.
Evaluating Vision-Language Models as a Zero-Shot Learning Alternative to You Only Look Once and Optical Character Recognition for Nigerian License Plate Recognition
Ahmadu Bello University（艾哈迈杜·贝罗大学）；Bayero University Kano（卡诺贝罗大学）
The University of Sydney（悉尼大学）；Baidu Inc.（百度公司）；Zhejiang University（浙江大学）；University of Chinese Academy of Sciences（中国科学院大学）；Shanghai AI Laboratory（上海人工智能实验室）
Vision-language models (VLMs) pre-trained on large-scale image-text pairs have demonstrated impressive transferability on various visual tasks. Transferring knowledge from such powerful VLMs is a promising direction for building effective video recognition models. However, current exploration in this field is still limited. We believe that the greatest value of pre-trained VLMs lies in building a bridge between visual and textual domains. In this paper, we propose a novel framework called BIKE, which utilizes the cross-modal bridge to explore bidirectional knowledge: i) We introduce the Video …
Texture or Semantics? Vision-Language Models Get Lost in Font Recognition
University of California, San Diego（加州大学圣地亚哥分校）；ByteDance（字节跳动）；The University of Queensland（昆士兰大学）；University of Southern California（南加州大学）；University at Buffalo（布法罗大学）；University of California, Merced（加州大学默塞德分校）
How Much Information Can a Vision Token Hold? A Scaling Law for Recognition Limits in VLMs
City University of Hong Kong（香港城市大学）；The Hong Kong Polytechnic University（香港理工大学）；The Hong Kong University of Science and Technology（香港理工大学）；Centre for Artificial Intelligence and Robotics, Chinese Academy of Sciences（中国科学院人工智能与机器人研究院）
Structured-Condensed Prompt Tuning in Vision-Language Models for Fine-grained Image Recognition
School of Information Science and Technology, Northwest University（西北大学信息科学与技术学院）；Laboratory of Intelligent Information Processing, Institute of Computing Technology（中国科学院计算技术研究所智能信息处理重点实验室）
Breaking Déjà Vu: Independent Auditing of Visual Place Recognition through Vision-Language Reasoning
School of Electronics and Computer Science, University of Southampton（南安普顿大学电子与计算机科学学院）；School of Electrical Engineering and Computer Science, Queensland University of Technology（昆士兰科技大学电气工程与计算机科学学院）；School of Computer Science and Electronic Engineering, University of Essex（埃塞克斯大学计算机科学与电子工程学院）
Are Vision Language Models Ready for Clinical Diagnosis? A 3D Medical Benchmark for Tumor-centric Visual Question Answering
Johns Hopkins University（约翰霍普金斯大学）；University of Bologna（博洛尼亚大学）；Istanbul Medipol University（伊斯坦布尔梅迪波尔大学）；Center for Biomolecular Nanotechnologies, Istituto Italiano di Tecnologia（意大利理工学院生物分子纳米技术中心）
