Vision-language models (VLMs) have made substantial progress across a wide range of visual question answering benchmarks, spanning visual reasoning, document understanding, and multimodal dialogue. These improvements are evident in a wide range of VLMs built on a variety of base models, alignment architectures, and training data. However, recent works show that these models trail behind in traditional image classification benchmarks, which test fine-grained visual knowledge. We test a large number of recent VLMs on fine-grained classification benchmarks and identify potential factors in the disconnect between fine-grained knowledge and other vision benchmarks. Through a series of ablation experiments, we find that using a better LLM improves all benchmark scores equally, while a better vision encoder disproportionately improves fine-grained classification performance. Furthermore, we find that the pretraining stage is also vital to fine-grained performance, particularly when the language model weights are unfrozen during pretraining. These insights pave the way for enhancing fine-grained visual understanding and vision-centric capabilities in VLMs.
Large-scale vision-language pre-training has shown impressive advances in a wide range of downstream tasks. Existing methods mainly model the cross-modal alignment by the similarity of the global representations of images and texts, or advanced cross-modal attention upon image and text features. However, they fail to explicitly learn the fine-grained semantic alignment between visual regions and textual phrases, as only global image-text alignment information is available. In this paper, we introduce LOUPE, a fine-grained semantically aLigned visiOn-langUage PrE-training framework, which learn…
FRISM: Fine-Grained Reasoning Injection via Subspace-Level Model Merging for Vision-Language Models
College of Future Information Technology, Fudan University, Shanghai, China（复旦大学未来信息科技学院，中国）；Shanghai Innovation Institute, China（上海创新研究院，中国）；The Chinese University of Hong Kong, China（香港中文大学，中国）；Harbin Institute of Technology, China（哈尔滨工业大学，中国）；Shanghai Artificial Intelligence Laboratory, China（上海人工智能实验室，中国）；Sun Yat-Sen University, Shenzhen, China（暨南大学深圳校区，中国）
CHIRP: A Fine-Grained Benchmark for Open-Ended Response Evaluation in Vision-Language Models
Mila - Quebec AI Institute（魁北克人工智能研究所）；Université de Montréal（蒙特利尔大学）；Tokyo Institute of Technology（东京技术大学）；McGill University（麦吉尔大学）；EleutherAI；University College London（伦敦大学学院）
Unveiling Chain of Step Reasoning for Vision-Language Models with Fine-grained Rewards
Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；School of Artificial Intelligence, University of Chinese Academy of Sciences（中国科学院大学人工智能学院）；Beijing Academy of Artificial Intelligence（北京人工智能研究院）
Benchmarking Large Vision-Language Models on Fine-Grained Image Tasks: A Comprehensive Evaluation
School of Computer Science and Engineering, School of Intelligence Science and Engineering and Key Laboratory of New Generation Artificial Intelligence Technology and Its Interdisciplinary Applications, Southeast University, China（东南大学计算机科学与工程学院、智能科学与工程学院及新一代人工智能技术及其跨学科应用关键实验室，中国）；Wangxuan Institute of Computer Technology, Peking University, China（北京大学王轩计算机技术研究所，中国）；University of Copenhagen, Denmark（丹麦哥本哈根大学）
State-Conditioned Visual Evidence Retrieval for Fine-Grained Perception in Document Vision-Language Models
College of Computer Science and Artificial Intelligence, Fudan University（复旦大学计算机科学与人工智能学院）；Shanghai Innovation Institute（上海创新研究院）；Shanghai Artificial Intelligence Laboratory（上海人工智能实验室）；ByteDance, LarkAI（字节跳动 LarkAI）；Shanghai Key Lab of Intelligent Information Processing（上海智能信息处理重点实验室）
Few-shot anomaly detection (FSAD) methods identify anomalous regions with few known normal samples. Most existing methods rely on the generalization ability of pre-trained vision-language models (VLMs) to recognize potentially anomalous regions through feature similarity between text descriptions and images. However, due to the lack of detailed textual descriptions, these methods can only pre-define image-level descriptions to match each visual patch token to identify potential anomalous regions, which leads to the semantic misalignment between image descriptions and patch-level visual anomali…
Benchmarking Large Vision-Language Models on Fine-Grained Image Tasks: From Evaluation to Diagnosis
School of Computer Science and Engineering, Southeast University, China（东南大学计算机科学与工程学院，中国）；Alibaba Group（阿里巴巴集团）；School of Computer Science and Engineering, School of Intelligence Science and Engineering, and Key Laboratory of New Generation Artificial Intelligence Technology and Its Interdisciplinary Applications, Southeast University, China（东南大学计算机科学与工程学院、智能科学与工程学院以及新一代人工智能技术及其交叉应用关键实验室，中国）；Wangxuan Institute of Computer Technology, National Key Laboratory for Multimedia Information Processing, Peking University, China（北京大学王轩计算机技术研究所、多媒体信息处理国家重点实验室，中国）；University of Copenhagen, Denmark（丹麦哥本哈根大学）
Vision Language Model for Interpretable and Fine-grained Detection of Safety Compliance in Diverse Workplaces
School of Mechanical, Aerospace, and Manufacturing Engineering（机械、航空航天与制造工程学院）；University of Connecticut（康涅狄格大学）；Department of Computer Science（计算机科学系）；University of California Irvine（加州大学欧文分校）
Analyzing Fine-Grained Alignment and Enhancing Vision Understanding in Multimodal Language Models
Department of Computer Science and Engineering, The Ohio State University（计算机科学与工程系，俄亥俄州立大学）；Translational Data Analytics Institute, The Ohio State University（转化数据分析研究所，俄亥俄州立大学）；Department of Biomedical Informatics, The Ohio State University（生物医学信息学系，俄亥俄州立大学）
Pix2Fact: When Vision Is Not Enough -- Benchmarking Fine-Grained VQA with Web Verification on High-Resolution Real-World Scenes
GADE Union (Global AI Data Experts Union)（GADE联盟（全球人工智能数据专家联盟））；Shanghai Jiao Tong University（上海交通大学）；Nanyang Technological University（南洋理工大学）；New York University（纽约大学）；Cambridge University（剑桥大学）；The University of Hong Kong（香港大学）
EviViT: Evidence-Adaptive Vision Transformers for Fine-Grained Perception
Tsinghua University（清华大学）；Peng Cheng Laboratory（鹏城实验室）；The Hong Kong University of Science and Technology（香港科技大学）；LMMs-Lab；Beijing Institute of Technology（北京理工大学）；University of New South Wales（新南威尔士大学）
EPIC-Bench: A Perception-Centric Benchmark for Fine-Grained Embodied Visual Grounding in Vision-Language Models
X-Humanoid；Fudan University（复旦大学）；University of Science and Technology of China（中国科学技术大学）；University of Manchester（曼彻斯特大学）；Monash University（墨尔本大学）；Celonis AI；University of New South Wales（新南威尔士大学）
DV-SFT: Direct Vision Supervision for Fine-Grained Visual Understanding
School of Computer Science and Technology, Beijing Institute of Technology（北京理工大学计算机科学与技术学院）；Zhongguancun Academy（中关村学院）；Beihang University（北航）；Zhongguancun Laboratory（中关村实验室）；Southeast Academy of Information Technology, Beijing Institute of Technology（北京理工大学信息科学技术东南学院）
Structured-Condensed Prompt Tuning in Vision-Language Models for Fine-grained Image Recognition
School of Information Science and Technology, Northwest University（西北大学信息科学与技术学院）；Laboratory of Intelligent Information Processing, Institute of Computing Technology（中国科学院计算技术研究所智能信息处理重点实验室）
fine-CLIP: Enhancing Zero-Shot Fine-Grained Surgical Action Recognition with Vision-Language Models
University of Strasbourg（斯特拉斯堡大学）；CNRS（法国国家科学研究中心）；INSERM（法国国家健康与医学研究院）；ICube（ICube实验室）；IHU Strasbourg（斯特拉斯堡大学医院研究所）；University Hospital of Strasbourg（斯特拉斯堡大学医院）
Vrije University of Amsterdam University of Amsterdam Centrum Wiskunde \& Informatic；University of Amsterdam National Institute of Informatics Centrum Wiskunde \& Informatic；Centrum Wiskunde \& Informatic Leiden University University College London；Vrije University of Amsterdam；Centrum Wiskunde \& Informatica Technische Universiteit Delft Guangzhou University；Vrije University of Amsterdam Centrum Wiskunde \& Informatic
Synthesize, Diagnose, and Optimize: Towards Fine-Grained Vision-Language Understanding
Fudan University（复旦大学）；NVIDIA（英伟达）；Shanghai Key Lab of Intell. Info. Processing（上海智能信息处理重点实验室）；Shanghai Collaborative Innovation Center of Intelligent Visual Computing（上海智能视觉计算协同创新中心）
PestVL-Net: Enabling Multimodal Pest Learning via Fine-grained Vision-Language Interaction
Institute of Intelligent Machines, Hefei Institutes of Physical Science, Chinese Academy of Sciences（智能机器研究所，合肥物理科学研究院，中国科学院）；University of Science and Technology of China（中国科学技术大学）；Zhongke Hefei Institute of Technology Innovation Engineering（中科合肥技术创新工程研究院）
Uncovering Intrinsic Capabilities: A Paradigm for Data Curation in Vision-Language Models
Harbin Institute of Technology, Shenzhen, China（哈尔滨工业大学（深圳））；Hong Kong Baptist University, China（香港 Baptist大学）；City University of Hong Kong, China（香港城市大学）
