Multi-modal Large Language Models (MLLMs) have made significant strides in expanding the capabilities of Large Language Models (LLMs) through the incorporation of visual perception interfaces. Despite the emergence of exciting applications and the availability of diverse instruction tuning data, existing approaches often rely on CLIP or its variants as the visual branch, and merely extract features from the deep layers. However, these methods lack a comprehensive analysis of the visual encoders in MLLMs. In this paper, we conduct an extensive investigation into the effectiveness of different vision encoders within MLLMs. Our findings reveal that the shallow layer features of CLIP offer particular advantages for fine-grained tasks such as grounding and region understanding. Surprisingly, the vision-only model DINO, which is not pretrained with text-image alignment, demonstrates promising performance as a visual branch within MLLMs. By simply equipping it with an MLP layer for alignment, DINO surpasses CLIP in fine-grained related perception tasks. Building upon these observations, we propose a simple yet effective feature merging strategy, named COMM, that integrates CLIP and DINO with Multi-level features Merging, to enhance the visual capabilities of MLLMs. We evaluate COMM through comprehensive experiments on a wide range of benchmarks, including image captioning, visual question answering, visual grounding, and object hallucination. Experimental results demonstrate the superior performance of COMM compared to existing methods, showcasing its enhanced visual capabilities within MLLMs.
AnchorScore: A CLIP-Based Diagnostic of MLLM Annotation Difficulty
School of Foreign Studies, Changsha University of Science and Technology（长沙理工大学外国语学院）；School of Education, Hunan Agricultural University（湖南农业大学教育学院）；School of Information and Intelligence, Hunan Agricultural University（湖南农业大学信息与智能科学学院）
ClimateVID -- Social Media Videos Analysis and Challenges Involved
University of Mannheim（曼海姆大学）；Technical University of Munich（慕尼黑技术大学）；Max-Planck-Institute for Informatics（马克斯·普朗克信息研究所）；Saarland Informatics Campus（萨尔兰州信息学校区）
COSMIC: Clique-Oriented Semantic Multi-space Integration for Robust CLIP Test-Time Adaptation
Shenzhen International Graduate School, Tsinghua University（清华大学深圳国际研究生院）；Shenzhen Technology University（深圳技术大学）；University of Science and Technology of China（中国科学技术大学）
VisualAD: Language-Free Zero-Shot Anomaly Detection via Vision Transformer
State Key Laboratory of Opto-Electronic Information Acquisition and Protection Technology, Anhui University, Hefei, China（光电信息采集与防护技术国家重点实验室，安徽大学，合肥，中国）；School of Artificial Intelligence, Anhui University, Hefei, China（人工智能学院，安徽大学，合肥，中国）；College of Intelligence Science and Technology, National University of Defense Technology, Changsha, China（智能科学与技术学院，国防科技大学，长沙，中国）
VLM-Aware Meta-Optic Front-End Design for Frozen Vision-Language Models
Department of Artificial Intelligence, Hanyang University（汉阳大学人工智能系）；School of Computational Science and Engineering, Georgia Institute of Technology（佐治亚理工学院计算科学与工程学院）；Department of Electronic Engineering, Hanyang University（汉阳大学电子工程系）
DINORANKCLIP: DINOv3 Distillation and Injection for Vision-Language Pretraining with High-Order Ranking Consistency
University of California, Los Angeles（加州大学洛杉矶分校）；Aimaikj（阿米亚克j）；HFIPS, Chinese Academy of Sciences（中国科学院HFIPS）；University of Science and Technology of China（中国科学技术大学）；Chinese Academy of Sciences（中国科学院）；National University of Defense Technology（国防科技大学）
