Spectral clustering is known as a powerful technique in unsupervised data analysis. The vast majority of approaches to spectral clustering are driven by a single modality, leaving the rich information in multi-modal representations untapped. Inspired by the recent success of vision-language pre-training, this paper enriches the landscape of spectral clustering from a single-modal to a multi-modal regime. Particularly, we propose Neural Tangent Kernel Spectral Clustering that leverages cross-modal alignment in pre-trained vision-language models. By anchoring the neural tangent kernel with positive nouns, i.e., those semantically close to the images of interest, we arrive at formulating the affinity between images as a coupling of their visual proximity and semantic overlap. We show that this formulation amplifies within-cluster connections while suppressing spurious ones across clusters, hence encouraging block-diagonal structures. In addition, we present a regularized affinity diffusion mechanism that adaptively ensembles affinity matrices induced by different prompts. Extensive experiments on \textbf{16} benchmarks -- including classical, large-scale, fine-grained and domain-shifted datasets -- manifest that our method consistently outperforms the state-of-the-art by a large margin.
Beijing National Research Center for Information Science and Technology (BNRist)（北京信息科学与技术国家研究中心（BNRist））；Tsinghua University（清华大学）
Clustering-based unsupervised domain adaptive (UDA) person re-identification (ReID) reduces exhaustive annotations. However, owing to unsatisfactory feature embedding and imperfect clustering, pseudo labels for target domain data inherently contain an unknown proportion of wrong ones, which would mislead feature learning. In this paper, we propose an approach named probabilistic uncertainty guided progressive label refinery (P$^2$LR) for domain adaptive person re-identification. First, we propose to model the labeling uncertainty with the probabilistic distance along with ideal single-peak dis…
Beihang University（北京航空航天大学）；Peking University（北京大学）；Shanghai AI Laboratory（上海人工智能实验室）
Vision Transformers (ViTs) have achieved impressive performance on various vision tasks, yet their generalization under distribution shifts (DS) is rarely understood. In this work, we comprehensively study the out-of-distribution (OOD) generalization of ViTs. For systematic investigation, we first present a taxonomy of DS. We then perform extensive evaluations of ViT variants under different DS and compare their generalization with Convolutional Neural Network (CNN) models. Important observations are obtained: 1) ViTs learn weaker biases on backgrounds and textures, while they are equipped wit…
University of Wisconsin-Madison（威斯康星大学麦迪逊分校）；Adobe（奥多比）；Google Research（谷歌研究院）
Recognizing out-of-distribution (OOD) samples is critical for machine learning systems deployed in the open world. The vast majority of OOD detection methods are driven by a single modality (e.g., either vision or language), leaving the rich information in multi-modal representations untapped. Inspired by the recent success of vision-language pre-training, this paper enriches the landscape of OOD detection from a single-modal to a multi-modal regime. Particularly, we propose Maximum Concept Matching (MCM), a simple yet effective zero-shot OOD detection method based on aligning visual features …
Faculty of Computer Science, University of Vienna（维也纳大学计算机科学学院）；UniVie Doctoral School Computer Science, University of Vienna（维也纳大学UniVie计算机科学博士学院）；LMU Munich（慕尼黑大学）；Munich Center for Machine Learning（慕尼黑机器学习中心）；Faculty of Philological and Cultural Studies, University of Vienna（维也纳大学语言与文化学院）
Vision Transformer (ViT) has become one of the most popular neural architectures due to its great scalability, computational efficiency, and compelling performance in many vision tasks. However, ViT has shown inferior performance to Convolutional Neural Network (CNN) on medical tasks due to its data-hungry nature and the lack of annotated medical data. In this paper, we pre-train ViTs on 266,340 chest X-rays using Masked Autoencoders (MAE) which reconstruct missing pixels from a small part of each image. For comparison, CNNs are also pre-trained on the same 266,340 X-rays using advanced self-s…
HEDGE: Hallucination Estimation via Dense Geometric Entropy for VQA with Vision-Language Models
Simula Metropolitan Center for Digital Engineering (SimulaMet)（Simula数字工程研究中心）；Oslo Metropolitan University (OsloMet)（奥斯陆 Metropolitan 大学）；Simula Research Laboratory（Simula研究实验室）
Recently, several clustering algorithms have been used to solve variety of problems from different discipline. This dissertation aims to address different challenging tasks in computer vision and pattern recognition by casting the problems as a clustering problem. We proposed novel approaches to solve multi-target tracking, visual geo-localization and outlier detection problems using a unified underlining clustering framework, i.e., dominant set clustering and its extensions, and presented a superior result over several state-of-the-art approaches.
Purdue University（普渡大学）；University of Seville（塞维利亚大学）；New York University（纽约大学）
We present the clustering learning technique applied to multi-layer feedforward deep neural networks. We show that this unsupervised learning technique can compute network filters with only a few minutes and a much reduced set of parameters. The goal of this paper is to promote the technique for general-purpose robotic vision systems. We report its use in static image datasets and object tracking datasets. We show that networks trained with clustering learning can outperform large networks trained for many hours on complex datasets.
Semantic Equitable Clustering: A Simple and Effective Strategy for Clustering Vision Tokens
MAIS & NLPR, Institute of Automation, Chinese Academy of Sciences, Beijing, China（自动化研究所，中国科学院，北京）；School of Artificial Intelligence, University of Chinese Academy of Sciences, Beijing, China（中国科学院大学人工智能学院，北京）
VLM4Cluster: Benchmarking Deep Clustering In the Era of Vision-Language Pre-training
University College London（伦敦大学学院）；The University of Queensland（昆士兰大学）；Southeast University（东南大学）；University of Science and Technology of China（中国科学技术大学）；Auckland University of Technology（奥克兰理工大学）
In this thesis, we present new schemes which leverage a constrained clustering method to solve several computer vision tasks ranging from image retrieval, image segmentation and co-segmentation, to person re-identification. In the last decades clustering methods have played a vital role in computer vision applications; herein, we focus on the extension, reformulation, and integration of a well-known graph and game theoretic clustering method known as Dominant Sets. Thus, we have demonstrated the validity of the proposed methods with extensive experiments which are conducted on several benchmar…
The University of Queensland（昆士兰大学）；University of Maryland（马里兰大学）
Reformulating computer vision problems over Riemannian manifolds has demonstrated superior performance in various computer vision applications. This is because visual data often forms a special structure lying on a lower dimensional space embedded in a higher dimensional space. However, since these manifolds belong to non-Euclidean topological spaces, exploiting their structures is computationally expensive, especially when one considers the clustering analysis of massive amounts of data. To this end, we propose an efficient framework to address the clustering problem on Riemannian manifolds. …
The Chinese University of Hong Kong（香港中文大学）；The University of Hong Kong（香港大学）；SenseTime Research（商汤科技研究院）；The University of Sydney（悉尼大学）
Vision transformers have achieved great successes in many computer vision tasks. Most methods generate vision tokens by splitting an image into a regular and fixed grid and treating each cell as a token. However, not all regions are equally important in human-centric vision tasks, e.g., the human body needs a fine representation with many tokens, while the image background can be modeled by a few tokens. To address this problem, we propose a novel Vision Transformer, called Token Clustering Transformer (TCFormer), which merges tokens by progressive clustering, where the tokens can be merged fr…
The University of Adelaide（阿德莱德大学）；DAMO Academy, Alibaba Group（阿里巴巴集团达摩院）；Northwestern Polytechnical University（西北工业大学）
Although Transformers have successfully transitioned from their language modelling origins to image-based applications, their quadratic computational complexity remains a challenge, particularly for dense prediction. In this paper we propose a content-based sparse attention method, as an alternative to dense self-attention, aiming to reduce the computation complexity while retaining the ability to model long-range dependencies. Specifically, we cluster and then aggregate key and value tokens, as a content-based method of reducing the total token count. The resulting clustered-token sequence re…
Self-supervised Multi-view Clustering in Computer Vision: A Survey
Inner Mongolia University of Technology（内蒙古工业大学）；OPPO Research Institute（OPPO研究院）；Haihe Laboratory of Information Technology Application Innovation（海河信息技术应用创新实验室）；Institute of Computing Technology, Chinese Academy of Sciences（中国科学院计算技术研究所）；InnoPeak Technology, Inc（InnoPeak科技公司）
The Chinese University of Hong Kong（香港中文大学）；Shenzhen Campus of Sun Yat-sen University（中山大学深圳校区）；Huawei Noah’s Ark Lab（华为诺亚方舟实验室）
To bridge the gap between supervised semantic segmentation and real-world applications that acquires one model to recognize arbitrary new concepts, recent zero-shot segmentation attracts a lot of attention by exploring the relationships between unseen and seen object categories, yet requiring large amounts of densely-annotated data with diverse base classes. In this paper, we propose a new open-world semantic segmentation pipeline that makes the first attempt to learn to segment semantic objects of various open-world categories without any efforts on dense annotations, by purely exploiting the…
TCFormer: Visual Recognition via Token Clustering Transformer
The Chinese University of Hong Kong（香港中文大学）；The University of Hong Kong（香港大学）；SenseTime Research（商汤科技研究院）；The University of Sydney（悉尼大学）；Shanghai AI Laboratory（上海人工智能实验室）；Centre for Perceptual and Interactive Intelligence Limited（感知与交互智能中心有限公司）
St. Petersburg Institute for Informatics and Automation of RAS（俄罗斯科学院圣彼得堡信息学与自动化研究所）
In this paper a hierarchical model for pixel clustering and image segmentation is developed. In the model an image is hierarchically structured. The original image is treated as a set of nested images, which are capable to reversibly merge with each other. An object is defined as a structural element of an image, so that, an image is regarded as a maximal object. The simulating of none-hierarchical optimal pixel clustering by hierarchical clustering is studied. To generate a hierarchy of optimized piecewise constant image approximations, estimated by the standard deviation of approximation fro…
South China Agricultural University（华南农业大学）；Sun Yat-sen University（中山大学）
Vision Transformer (ViT) has shown its advantages over the convolutional neural network (CNN) with its ability to capture global long-range dependencies for visual representation learning. Besides ViT, contrastive learning is another popular research topic recently. While previous contrastive learning works are mostly based on CNNs, some recent studies have attempted to combine ViT and contrastive learning for enhanced self-supervised learning. Despite the considerable progress, these combinations of ViT and contrastive learning mostly focus on the instance-level contrastiveness, which often o…
FTCFormer: Fuzzy Token Clustering Transformer for Image Classification
School of Advanced Technology, Xi'an Jiaotong-Liverpool University, Suzhou, China（西安交通大学利物浦大学先进技术学院）；Department of Mathematical Sciences, University of Liverpool, Liverpool, United Kingdom（利物浦大学数学科学系）；Department of Computer Science, University of Liverpool, Liverpool, United Kingdom（利物浦大学计算机科学系）
Moving object detection is important in computer vision. Event-based cameras are bio-inspired cameras that work by mimicking the working of the human eye. These cameras have multiple advantages over conventional frame-based cameras, like reduced latency, HDR, reduced motion blur during high motion, low power consumption, etc. In spite of these advantages, event-based cameras are noise-sensitive and have low resolution. Moreover, the task of moving object detection in these cameras is difficult, as event-based sensors lack useful visual features like texture and color. In this paper, we investi…
Vision Transformers for Zero-Shot Clustering of Animal Images: A Comparative Benchmarking Study
Department of Chemistry and Bioscience（化学与生物科学系）；Aalborg University（奥尔堡大学）；Department of Architecture, Design and Media Technology（建筑、设计与媒体技术系）；Visual Analysis and Perception Lab（视觉分析与感知实验室）
Vision-and-Language Navigation in Continuous Environments (VLN-CE) is a navigation task that requires an agent to follow a language instruction in a realistic environment. The understanding of environments is a crucial part of the VLN-CE task, but existing methods are relatively simple and direct in understanding the environment, without delving into the relationship between language instructions and visual environments. Therefore, we propose a new environment representation in order to solve the above problems. First, we propose an Environment Representation Graph (ERG) through object detecti…
FA: Forced Prompt Learning of Vision-Language Models for Out-of-Distribution Detection
Sun Yat-sen University（中山大学）；Peng Cheng Laboratory（鹏城实验室）；Hong Kong University of Science and Technology (Guangzhou)（香港科技大学（广州））；Key Laboratory of Machine Intelligence and Advanced Computing（人工智能与先进计算重点实验室）
本文提出基于 CLIP 的强制提示学习框架 FA，通过充分挖掘 ID 类别语义并引入强制系数，在无需外部辅助数据且参数量与 CoOp 相同的情况下提升 OOD 检测性能。
Deep Convolutional Neural Networks (CNN) has achieved significant success in computer vision field. However, the high computational cost of the deep complex models prevents the deployment on edge devices with limited memory and computational resource. In this paper, we proposed a novel filter pruning for convolutional neural networks compression, namely spectral clustering filter pruning with soft self-adaption manners (SCSP). We first apply spectral clustering on filters layer by layer to explore their intrinsic connections and only count on efficient groups. By self-adaption manners, the pru…
SAC-ViT: Semantic-Aware Clustering Vision Transformer with Early Exit
Faculty of Computing, Harbin Institute of Technology（哈尔滨工业大学计算机学院）；Swiss Data Science Center（瑞士数据科学中心）；School of Cyber Engineering, Xidian University（西安电子科技大学网络工程学院）
University of Central Florida（中佛罗里达大学）；Purdue University（普渡大学）
Classification and clustering have been studied separately in machine learning and computer vision. Inspired by the recent success of deep learning models in solving various vision problems (e.g., object recognition, semantic segmentation) and the fact that humans serve as the gold standard in assessing clustering algorithms, here, we advocate for a unified treatment of the two problems and suggest that hierarchical frameworks that progressively build complex patterns on top of the simpler ones (e.g., convolutional neural networks) offer a promising solution. We do not dwell much on the learni…
Moving object detection has been a central topic of discussion in computer vision for its wide range of applications like in self-driving cars, video surveillance, security, and enforcement. Neuromorphic Vision Sensors (NVS) are bio-inspired sensors that mimic the working of the human eye. Unlike conventional frame-based cameras, these sensors capture a stream of asynchronous 'events' that pose multiple advantages over the former, like high dynamic range, low latency, low power consumption, and reduced motion blur. However, these advantages come at a high cost, as the event camera data typical…
