Vision-language models (VLMs) like CLIP have been cherished for their ability to perform zero-shot visual recognition on open-vocabulary concepts. This is achieved by selecting the object category whose textual representation bears the highest similarity with the query image. While successful in some domains, this method struggles with identifying fine-grained entities as well as generalizing to unseen concepts that are not captured by the training distribution. Recent works attempt to mitigate these challenges by integrating category descriptions at test time, albeit yielding modest improvements. We attribute these limited gains to a fundamental misalignment between image and description representations, which is rooted in the pretraining structure of CLIP. In this paper, we propose GRAIN, a new pretraining strategy aimed at aligning representations at both fine and coarse levels simultaneously. Our approach learns to jointly ground textual descriptions in image regions along with aligning overarching captions with global image representations. To drive this pre-training, we leverage frozen Multimodal Large Language Models (MLLMs) to derive large-scale synthetic annotations. We demonstrate the enhanced zero-shot performance of our model compared to current state-of-the art methods across 11 diverse image classification datasets. Additionally, we introduce Products-2023, a newly curated, manually labeled dataset featuring novel concepts, and showcase our model's ability to recognize these concepts by benchmarking on it. Significant improvements achieved by our model on other downstream tasks like retrieval further highlight the superior quality of representations learned by our approach. Code available at https://github.com/shaunak27/grain-clip .
Solving Zero-Shot 3D Visual Grounding as Constraint Satisfaction Problems
Bernoulli Institute for Mathematics, Computer Science and Artificial Intelligence, University of Groningen（格罗宁根大学伯努利学院）；Computer Vision for Human-Computer Interaction Lab (cv:hci), Karlsruhe Institute of Technology（卡尔斯鲁厄理工学院人机交互计算机视觉实验室）
VLM-Grounder: A VLM Agent for Zero-Shot 3D Visual Grounding
The Chinese University of Hong Kong（香港中文大学）；Shanghai AI Laboratory（上海人工智能实验室）；Zhejiang University（浙江大学）；Centre for Perceptual and Interactive Intelligence（感知与交互智能中心）
Visual Programming for Zero-shot Open-Vocabulary 3D Visual Grounding
The Future Network of Intelligence Institute（未来智能网络研究院）；The Chinese University of Hong Kong (Shenzhen)（香港中文大学（深圳））；School of Science and Engineering, The Chinese University of Hong Kong (Shenzhen)（香港中文大学（深圳）理工学院）；IHPC, A*STAR, Singapore（新加坡科研局智能系统研究所）；The University of Hong Kong（香港大学）
Multiple Consistent 2D-3D Mappings for Robust Zero-Shot 3D Visual Grounding
Hangzhou Dianzi University（杭州电子科技大学）；Zhejiang University（浙江大学）；Alibaba Group（阿里巴巴集团）；Institute for Clarity in Documentation（文档清晰研究所）；Inria Paris-Rocquencourt（Inria巴黎-鲁克蒙特研究所）；Rajiv Gandhi University（拉贾·甘地大学）；Tsinghua University（清华大学）；Palmer Research Laboratories（帕勒实验室）
SeqVLM: Proposal-Guided Multi-View Sequences Reasoning via VLM for Zero-Shot 3D Visual Grounding
School of Informatics, Xiamen University（厦门大学信息学院）；School of Computer Science, Nanjing University（南京大学计算机科学学院）；School of Computer Science and Technology, East China Normal University（华东师范大学计算机科学与技术学院）；Key Laboratory of Multimedia Trusted Perception and Efficient Computing, Ministry of Education of China, Xiamen University（教育部多媒体可信感知与高效计算重点实验室，厦门大学）
CPT: Colorful Prompt Tuning for Pre-trained Vision-Language Models
Tsinghua University（清华大学）；Beijing National Research Center for Information Science and Technology（北京信息科学与技术国家研究中心）；National University of Singapore（新加坡国立大学）
Training-Free Open-Vocabulary Visual Grounding for Remote Sensing Images and Videos
School of Computer Science and Technology, Xidian University（西安电子科技大学计算机科学与技术学院）；Interdisciplinary Institute of Artificial Intelligence, Xidian University（西安电子科技大学跨学科人工智能研究院）；School of Artificial Intelligence, Xidian University（西安电子科技大学人工智能学院）；Northwest Institute of Nuclear Technology（西北核技术研究所）；School of Physics and Information Engineering, Fuzhou University（福州大学物理与信息工程学院）
Reasoning3D -- Grounding and Reasoning in 3D: Fine-Grained Zero-Shot Open-Vocabulary 3D Reasoning Part Segmentation via Large Vision-Language Models
KOKONI, Moxin (Huzhou) Tech. Co., LTD.（杭州魔芯科技有限公司）；College of Computer Science and Technology, Zhejiang University（浙江大学计算机科学与技术学院）；School of Information Engineering, Huzhou University（湖州师范学院信息工程学院）；Singapore University of Technology and Design（新加坡科技设计大学）；School of Information Science and Technology, University of Science and Technology of China（中国科学技术大学信息科学技术学院）；School of Software Technology, Zhejiang University（浙江大学软件学院）
VideoGEM: Training-free Action Grounding in Videos
Goethe University Frankfurt（法兰克福大学）；Tuebingen AI Center（蒂宾根人工智能中心）；University of Tuebingen（蒂宾根大学）；MPI for Informatics（马克斯·普朗克信息学研究所）；MIT-IBM Watson AI Lab（麻省理工学院-IBM沃森人工智能实验室）
From What to Which: Decoding Modifier Grounding in Frozen MLLMs
AI for Good (AIGO)（人工智能促进美好（AIGO））；Istituto Italiano di Tecnologia（意大利技术研究院）；University of Siena（锡耶纳大学）；University of Genoa（热那亚大学）；University of Verona（维罗纳大学）
ZING-3D: Zero-shot Incremental 3D Scene Graphs via Vision-Language Models
Birla Institute of Technology and Science Pilani, K.K Birla Goa Campus（比拉理工学院和科学学院，比拉戈阿校园）；Department of Mechanical Engineering, College of Design and Engineering, National University of Singapore（设计与工程学院机械工程系，新加坡国立大学）
