Recent research has shown that CLIP models struggle with visual reasoning tasks that require grounding compositionality, understanding spatial relationships, or capturing fine-grained details. One natural hypothesis is that the CLIP vision encoder does not embed essential information for these tasks. However, we find that this is not always the case: The encoder gathers query-relevant visual information, while CLIP fails to extract it. In particular, we show that another branch of Vision-Language Models (VLMs), Generative Multimodal Large Language Models (MLLMs), achieve significantly higher accuracy than CLIP in many of these tasks using the same vision encoder and weights, indicating that these Generative MLLMs perceive more -- as they extract and utilize visual information more effectively. We conduct a series of controlled experiments and reveal that their success is attributed to multiple key design choices, including patch tokens, position embeddings, and prompt-based weighting. On the other hand, enhancing the training data alone or applying a stronger text encoder does not suffice to solve the task, and additional text tokens offer little benefit. Interestingly, we find that fine-grained visual reasoning is not exclusive to generative models trained by an autoregressive loss: When converted into CLIP-like encoders by contrastive finetuning, these MLLMs still outperform CLIP under the same cosine similarity-based evaluation protocol. Our study highlights the importance of VLM architectural choices and suggests directions for improving the performance of CLIP-like contrastive VLMs.
AnchorScore: A CLIP-Based Diagnostic of MLLM Annotation Difficulty
School of Foreign Studies, Changsha University of Science and Technology（长沙理工大学外国语学院）；School of Education, Hunan Agricultural University（湖南农业大学教育学院）；School of Information and Intelligence, Hunan Agricultural University（湖南农业大学信息与智能科学学院）
Jailbreaking Multimodal Large Language Models using Multi-Clip Video
Department of Applied Artificial Intelligence, Sungkyunkwan University（应用人工智能系，成均馆大学）；Department of Human-Artificial Intelligence Interaction, Sungkyunkwan University（人机交互系，成均馆大学）
Adaptive Global and Fine-Grained Perceptual Fusion for MLLM Embeddings Compatible with Hard Negative Amplification
State Key Lab of General AI, School of Intelligence Science and Technology, Peking University（人工智能国家重点实验室，智能科学与技术学院，北京大学）；Institute for Artificial Intelligence, Peking University（人工智能研究院，北京大学）
University of California, Berkeley（加州大学伯克利分校）；University of California, Los Angeles（加州大学洛杉矶分校）；University of North Carolina at Chapel Hill（北卡罗来纳大学教堂山分校）
