Vision-Language Models (VLMs) have demonstrated strong capability in a wide range of tasks such as visual recognition, document parsing, and visual grounding. Nevertheless, recent work shows that while VLMs often manage to capture the correct image region corresponding to the question, they do not necessarily produce the correct answers. In this work, we demonstrate that this misalignment could be attributed to suboptimal information flow within VLMs, where text tokens distribute too much attention to irrelevant visual tokens, leading to incorrect answers. Based on the observation, we show that modulating the information flow during inference can improve the perception capability of VLMs. The idea is that text tokens should only be associated with important visual tokens during decoding, eliminating the interference of irrelevant regions. To achieve this, we propose a token dynamics-based method to determine the importance of visual tokens, where visual tokens that exhibit distinct activation patterns during different decoding stages are viewed as important. We apply our approach to representative open-source VLMs and evaluate on various datasets, including visual question answering, visual grounding and counting, optical character recognition, and object hallucination. The results show that our approach significantly improves the performance of baselines. Project page: https://cxliu0.github.io/AIF/.
Seeing is Believing: Aligning Prompt Rewriting with Visual Anchors for Text-to-Image Generation
Peking University（北京大学）；Tencent（腾讯）；Dalian University of Technology（大连理工大学）；Nanyang Technological University（南洋理工大学）；University of Cambridge（剑桥大学）；Zhejiang University（浙江大学）
On Aligning Hierarchical Standardized Embedding for Audio-visual Generalized Zero-shot Learning
Southeast University（东南大学）；The University of Hong Kong（香港大学）；Beijing Institute of Technology（北京理工大学）；Key Laboratory of New Generation Artificial Intelligence Technology and Its Interdisciplinary Applications (Southeast University), Ministry of Education（新一代人工智能技术及其跨学科应用重点实验室（东南大学），教育部）；School of Computer Science and Engineering, Southeast University（东南大学计算机科学与工程学院）
BLINK: Multimodal Large Language Models Can See but Not Perceive
University of Pennsylvania（宾夕法尼亚大学）；University of Washington（华盛顿大学）；Allen Institute for AI（艾伦人工智能研究所）；University of California, Davis（加利福尼亚大学戴维斯分校）；Columbia University（哥伦比亚大学）
