Spatial intelligence in vision-language models (VLMs) attracts research interest with the practical demand to reason in the 3D world.Despite promising results, most existing methods follow the conventional 2D pipeline in VLMs and use pixel-aligned representations for the vision modality. However, correspondence-based models with implicit 3D scene understanding often fail to achieve spatial consistency, and representation-based models with 3D geometric priors lack efficiency in vision sequence serialization. To address this, we propose a Proxy3D method with compact yet comprehensive 3D proxy representations for the vision modality. Given only video frames as input, we employ semantic and geometric encoders to extract scene features and then perform their semantic-aware clustering to obtain a set of proxies in the 3D space. For representation alignment, we further curate the SpaceSpan dataset and apply multi-stage training to adopt the proposed 3D proxy representations with the VLM. When using shorter sequences for vision information, our method achieves competitive or state-of-the-art performance in 3D visual question answering, visual grounding and general spatial intelligence benchmarks.
GTA-Net: Cooperative Game Theory for Vision-Language Alignment in Chest X-Ray Report Generation
Department of Computer Science, Rhineland-Palatinate Technical University of Kaiserslautern-Landau（莱茵兰-普法尔茨凯泽斯劳滕-兰道工业大学计算机科学系）；German Research Center for Artificial Intelligence (DFKI)（德国人工智能研究中心）；IntelligentX GmbH
Supervised Classification Heads as Semantic Prototypes: Unlocking Vision-Language Alignment via Weight Recycling
Department of Computer Science and Artificial Intelligence, DaSCI Institute, University of Granada, Granada, Spain（计算机科学与人工智能系，DaSCI研究所，格拉纳达大学，格拉纳达，西班牙）；Department of Mathematics ``Tullio Levi-Civita'', University of Padova, Padova, Italy（托里利-西维塔数学系，帕多瓦大学，帕多瓦，意大利）
Contrastive vision-language models (e.g. CLIP) are typically created by updating all the parameters of a vision model and language model through contrastive training. Can such models be created by a small number of parameter updates to an already-trained language model and vision model? The literature describes techniques that can create vision-language models by updating a small number of parameters in a language model, but these require already aligned visual representations and are non-contrastive, hence unusable for latency-sensitive applications such as neural search. We explore the feasi…
Efficient Medical Vision-Language Alignment Through Adapting Masked Vision Models
School of Informatics, Xiamen University（厦门大学信息学院）；Center for Smart Health, School of Nursing, The Hong Kong Polytechnic University（香港理工大学护理学院智能健康中心）；Department of Biomedical Informatics, Harvard Medical School（哈佛医学院生物医学信息学系）；Department of Radiology, Zhongshan Hospital (Xiamen), Fudan University（复旦大学中山医院放射科）；National Institute for Data Science in Health and Medicine（医学与数据科学国家研究院）；Department of Computer Science, School of Informatics, Xiamen University（厦门大学信息学院计算机科学系）
Analyzing Fine-Grained Alignment and Enhancing Vision Understanding in Multimodal Language Models
Department of Computer Science and Engineering, The Ohio State University（计算机科学与工程系，俄亥俄州立大学）；Translational Data Analytics Institute, The Ohio State University（转化数据分析研究所，俄亥俄州立大学）；Department of Biomedical Informatics, The Ohio State University（生物医学信息学系，俄亥俄州立大学）
Large Vision-Language Model Alignment and Misalignment: A Survey Through the Lens of Explainability
Northwestern University（西北大学）；New Jersey Institute of Technology（新泽西理工学院）；University of Bristol（布里斯托大学）；Cisco Research（思科研究）；University of Illinois Chicago（伊利诺伊大学芝加哥分校）
Vision and Language Pretraining has become the prevalent approach for tackling multimodal downstream tasks. The current trend is to move towards ever larger models and pretraining datasets. This computational headlong rush does not seem reasonable in the long term to move toward sustainable solutions, and de facto excludes academic laboratories with limited resources. In this work, we propose a new framework, dubbed ViCHA, that efficiently exploits the input data to boost the learning by: (a) a new hierarchical cross-modal alignment loss, (b) new self-supervised scheme based on masked image mo…
Vision-R1: Evolving Human-Free Alignment in Large Vision-Language Models via Vision-Guided Reinforcement Learning
Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；University of Chinese Academy of Sciences（中国科学院大学）；Peng Cheng Laboratory（鹏城实验室）；Wuhan AI Research（武汉人工智能研究院）
TinyAlign: Boosting Lightweight Vision-Language Models by Mitigating Modal Alignment Bottlenecks
Beijing Advanced Innovation Center for Future Blockchain and Privacy Computing（北京未来区块链与隐私计算先进创新中心）；Beihang University（北京航空航天大学）；Hangzhou International Innovation Institute（杭州国际创新研究院）；Xreal；Renmin University（中国人民大学）；Peking University（北京大学）；Beijing Academy of Blockchain and Edge Computing (BABEC)（北京区块链与边缘计算研究院）
Recently, several clustering algorithms have been used to solve variety of problems from different discipline. This dissertation aims to address different challenging tasks in computer vision and pattern recognition by casting the problems as a clustering problem. We proposed novel approaches to solve multi-target tracking, visual geo-localization and outlier detection problems using a unified underlining clustering framework, i.e., dominant set clustering and its extensions, and presented a superior result over several state-of-the-art approaches.
Purdue University（普渡大学）；University of Seville（塞维利亚大学）；New York University（纽约大学）
We present the clustering learning technique applied to multi-layer feedforward deep neural networks. We show that this unsupervised learning technique can compute network filters with only a few minutes and a much reduced set of parameters. The goal of this paper is to promote the technique for general-purpose robotic vision systems. We report its use in static image datasets and object tracking datasets. We show that networks trained with clustering learning can outperform large networks trained for many hours on complex datasets.
Semantic Equitable Clustering: A Simple and Effective Strategy for Clustering Vision Tokens
MAIS & NLPR, Institute of Automation, Chinese Academy of Sciences, Beijing, China（自动化研究所，中国科学院，北京）；School of Artificial Intelligence, University of Chinese Academy of Sciences, Beijing, China（中国科学院大学人工智能学院，北京）
VLM4Cluster: Benchmarking Deep Clustering In the Era of Vision-Language Pre-training
University College London（伦敦大学学院）；The University of Queensland（昆士兰大学）；Southeast University（东南大学）；University of Science and Technology of China（中国科学技术大学）；Auckland University of Technology（奥克兰理工大学）
In this thesis, we present new schemes which leverage a constrained clustering method to solve several computer vision tasks ranging from image retrieval, image segmentation and co-segmentation, to person re-identification. In the last decades clustering methods have played a vital role in computer vision applications; herein, we focus on the extension, reformulation, and integration of a well-known graph and game theoretic clustering method known as Dominant Sets. Thus, we have demonstrated the validity of the proposed methods with extensive experiments which are conducted on several benchmar…
Adaptive Cortically Constrained EEG-Vision Alignment for Zero-Shot Brain-to-Image Retrieval
Chongqing University of Posts and Telecommunications（重庆邮电大学）；Shanghai Jiaotong University（上海交通大学）；Chongqing Normal University（重庆师范大学）；South China University of Technology（华南理工大学）
Do Vision Transformers See Like Humans? Evaluating their Perceptual Alignment
Image Processing Lab, Universidad de Valencia（瓦伦西亚大学图像处理实验室）；Center for Biomaterials and Tissue Engineering Universitat Politecnica de Valencia（多尔多涅理工大学生物材料与组织工程中心）
DiscoVLA: Discrepancy Reduction in Vision, Language, and Alignment for Parameter-Efficient Video-Text Retrieval
School of Software（软件学院）；BNRist；Department of Automation, Tsinghua University（自动化系，清华大学）；Hangzhou Zhuoxi Institute of Brain and Intelligence（杭州卓溪脑科学与智能研究院）；GRG Banking Equipment Co., Ltd.（GRG银行设备有限公司）；South China University of Technology（华南理工大学）
