This paper addresses the task of zero-shot image classification. The key contribution of the proposed approach is to control the semantic embedding of images -- one of the main ingredients of zero-shot learning -- by formulating it as a metric learning problem. The optimized empirical criterion associates two types of sub-task constraints: metric discriminating capacity and accurate attribute prediction. This results in a novel expression of zero-shot learning not requiring the notion of class in the training phase: only pairs of image/attributes, augmented with a consistency indicator, are given as ground truth. At test time, the learned model can predict the consistency of a test image with a given set of attributes , allowing flexible ways to produce recognition inferences. Despite its simplicity, the proposed approach gives state-of-the-art results on four challenging datasets used for zero-shot recognition evaluation.
University of Illinois at Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）
In this paper, we explore how to utilize pre-trained language model to perform few-shot text classification where only a few annotated examples are given for each class. Since using traditional cross-entropy loss to fine-tune language model under this scenario causes serious overfitting and leads to sub-optimal generalization of model, we adopt supervised contrastive learning on few labeled data and consistency-regularization on vast unlabeled data. Moreover, we propose a novel contrastive consistency to further boost model performance and refine sentence representation. After conducting exten…
Improving Few-Shot Learning through Multi-task Representation Learning Theory
Université Paris-Saclay（巴黎-萨克雷大学）；CEA（法国原子能委员会）；Université de Lyon（里昂大学）；UJM-Saint-Etienne（圣埃蒂安大学）；CNRS（法国国家科学研究中心）；IOGS（法国光学研究院）；Institut Universitaire de France (IUF)（法国大学研究院）
Deep learning classifiers are assisting humans in making decisions and hence the user's trust in these models is of paramount importance. Trust is often a function of constant behavior. From an AI model perspective it means given the same input the user would expect the same output, especially for correct outputs, or in other words consistently correct outputs. This paper studies a model behavior in the context of periodic retraining of deployed models where the outputs from successive generations of the models might not agree on the correct labels assigned to the same input. We formally defin…
Improving Graph Few-shot Learning with Hyperbolic Space and Denoising Diffusion
Key Laboratory of Symbolic Computation and Knowledge Engineering of the Ministry of Education, College of Computer Science and Technology, Jilin University（教育部符号计算与知识工程重点实验室，计算机科学与技术学院，吉林大学）；School of Mathematical and Computer Sciences, Heriot-Watt University（数学与计算机科学学院，赫瑞-沃顿大学）；Department of Information Engineering and Computer Science, University of Trento（信息工程与计算机科学系，特伦托大学）
ESPT: A Self-Supervised Episodic Spatial Pretext Task for Improving Few-Shot Learning
School of Computer Science and Artificial Intelligence, Wuhan University of Technology（武汉理工大学计算机科学与人工智能学院）；Sanya Science and Education Innovation Park, Wuhan University of Technology（武汉理工大学三亚科教创新园）；Hainan Yazhou Bay Seed Laboratory（海南崖州湾种子实验室）；Shanghai Artificial Intelligence Laboratory（上海人工智能实验室）
In image recognition, there are many cases where training samples cannot cover all target classes. Zero-shot learning (ZSL) utilizes the class semantic information to classify samples of the unseen categories that have no corresponding samples contained in the training set. In this paper, we propose an end-to-end framework, called Global Semantic Consistency Network (GSC-Net for short), which makes complete use of the semantic information of both seen and unseen classes, to support effective zero-shot learning. We also adopt a soft label embedding loss to further exploit the semantic relations…
Huawei（华为）；McMaster University（麦克马斯特大学）；University of Manitoba（曼尼托巴大学）
In this work, we present the winning solution for ORBIT Few-Shot Video Object Recognition Challenge 2022. Built upon the ProtoNet baseline, the performance of our method is improved with three effective techniques. These techniques include the embedding adaptation, the uniform video clip sampler and the invalid frame detection. In addition, we re-factor and re-implement the official codebase to encourage modularity, compatibility and improved performance. Our implementation accelerates the data loading in both training and testing.
University of Exeter（埃克塞特大学）；University of Tübingen（图宾根大学）
Low-shot sketch-based image retrieval is an emerging task in computer vision, allowing to retrieve natural images relevant to hand-drawn sketch queries that are rarely seen during the training phase. Related prior works either require aligned sketch-image pairs that are costly to obtain or inefficient memory fusion layer for mapping the visual information to a semantic space. In this paper, we address any-shot, i.e. zero-shot and few-shot, sketch-based image retrieval (SBIR) tasks, where we introduce the few-shot setting for SBIR. For solving these tasks, we propose a semantically aligned pair…
University of Montreal（蒙特利尔大学）；Mila – Quebec Artificial Intelligence Institute（米拉——魁北克人工智能研究所）
Recent work on few-shot learning \cite{tian2020rethinking} showed that quality of learned representations plays an important role in few-shot classification performance. On the other hand, the goal of self-supervised learning is to recover useful semantic information of the data without the use of class labels. In this work, we exploit the complementarity of both paradigms via a multi-task framework where we leverage recent self-supervised methods as auxiliary tasks. We found that combining multiple tasks is often beneficial, and that solving them simultaneously can be done efficiently. Our re…
Few-Shot Learning (FSL) aims to improve a model's generalization capability in low data regimes. Recent FSL works have made steady progress via metric learning, meta learning, representation learning, etc. However, FSL remains challenging due to the following longstanding difficulties. 1) The seen and unseen classes are disjoint, resulting in a distribution shift between training and testing. 2) During testing, labeled data of previously unseen classes is sparse, making it difficult to reliably extrapolate from labeled support examples to unlabeled query examples. To tackle the first challenge…
Improving Consistency Models with Generator-Augmented Flows
Criteo AI Lab, Paris, France（Criteo AI实验室，法国巴黎）；Reasoning (AI/R) Laboratory, Korea Institute of Science（推理（AI/R）实验室，韩国科学技术院）；Robot Department, University of Science（机器人部门，科学大学）
Visual Prompting for One-shot Controllable Video Editing without Inversion
Singapore University of Technology and Design（新加坡科技设计大学）；Wuhan University（武汉大学）；Institute for Infocomm Research, Agency for Science, Technology and Research, Singapore（新加坡资讯与通信研究院）
InfinityStory: Unlimited Video Generation with World Consistency and Character-Aware Shot Transitions
Adobe Research（Adobe研究院）；KAUST（卡塔尔科技大学）；University of Oregon（俄勒冈大学）；University of Memphis（密苏里大学孟菲斯分校）；Johns Hopkins University（约翰霍普金斯大学）；Meta AI；Texas A&M University（德克萨斯农工大学）；Dolby Labs（杜比实验室）；Virginia Tech（弗吉尼亚理工大学）；Cisco（思科）；University of Maryland, College Park（马里兰大学学院市分校）
In this work, we propose to use out-of-distribution samples, i.e., unlabeled samples coming from outside the target classes, to improve few-shot learning. Specifically, we exploit the easily available out-of-distribution samples to drive the classifier to avoid irrelevant features by maximizing the distance from prototypes to out-of-distribution samples while minimizing that of in-distribution samples (i.e., support, query data). Our approach is simple to implement, agnostic to feature extractors, lightweight without any additional cost for pre-training, and applicable to both inductive and tr…
Computer Vision Center（计算机视觉中心）；Autonomous University of Barcelona（巴塞罗那自治大学）；Amsterdam Machine Learning Lab（阿姆斯特丹机器学习实验室）；University of Amsterdam（阿姆斯特丹大学）
Zero-shot sketch-based image retrieval (SBIR) is an emerging task in computer vision, allowing to retrieve natural images relevant to sketch queries that might not been seen in the training phase. Existing works either require aligned sketch-image pairs or inefficient memory fusion layer for mapping the visual information to a semantic space. In this work, we propose a semantically aligned paired cycle-consistent generative (SEM-PCYC) model for zero-shot SBIR, where each branch maps the visual information to a common semantic space via an adversarial training. Each of these branches maintains …
Consistency-Driven Calibration and Matching for Few-Shot Class-Incremental Learning
School of Automation, Central South University, China（中南大学自动化学院）；Big Data Institute, Central South University, China（中南大学大数据研究院）；School of Computer Science, Faculty of Engineering, The University of Sydney, Australia（悉尼大学工程学院计算机科学系）
Labeling large image datasets with attributes such as facial age or object type is tedious and sometimes infeasible. Supervised machine learning methods provide a highly accurate solution, but require manual labels which are often unavailable. Zero-shot models (e.g., CLIP) do not require manual labels but are not as accurate as supervised ones, particularly when the attribute is numeric. We propose a new approach, CLIPPR (CLIP with Priors), which adapts zero-shot models for regression and classification on unlabelled datasets. Our method does not use any annotated images. Instead, we assume a …
The lack of a large amount of training data has always been the constraining factor in solving a lot of problems in machine learning, making One Shot Learning one of the most intriguing ideas in machine learning. It aims to learn information about object categories from one, or only a few training examples. This process of learning in deep learning is usually accomplished by proper objective function, i.e; loss function and embeddings extraction i.e; architecture. In this paper, we discussed about metrics based deep learning architectures for one shot learning such as Siamese neural networks a…
Korea Advanced Institute of Science and Technology (KAIST)（韩国科学技术院）
Few-shot learning often involves metric learning-based classifiers, which predict the image label by comparing the distance between the extracted feature vector and class representations. However, applying global pooling in the backend of the feature extractor may not produce an embedding that correctly focuses on the class object. In this work, we propose a novel framework that generates class representations by extracting features from class-relevant regions of the images. Given only a few exemplary images with image-level labels, our framework first localizes the class objects by spatially …
Semi-supervised learning (SSL) is a key approach toward more data-efficient machine learning by jointly leverage both labeled and unlabeled data. We propose AlphaMatch, an efficient SSL method that leverages data augmentations, by efficiently enforcing the label consistency between the data points and the augmented data derived from them. Our key technical contribution lies on: 1) using alpha-divergence to prioritize the regularization on data with high confidence, achieving a similar effect as FixMatch but in a more flexible fashion, and 2) proposing an optimization-based, EM-like algorithm t…
Learning with Unreliability: Fast Few-shot Voxel Radiance Fields with Relative Geometric Consistency
Singapore Management University（新加坡管理大学）；South China University of Technology（华南理工大学）；Nanjing University of Science and Technology（南京理工大学）；Cardiff University（卡迪夫大学）
Toyota Technological Institute at Chicago（芝加哥丰田技术研究所）；Google（谷歌）
We leverage unsupervised learning of depth, egomotion, and camera intrinsics to improve the performance of single-image semantic segmentation, by enforcing 3D-geometric and temporal consistency of segmentation masks across video frames. The predicted depth, egomotion, and camera intrinsics are used to provide an additional supervision signal to the segmentation model, significantly enhancing its quality, or, alternatively, reducing the number of labels the segmentation model needs. Our experiments were performed on the ScanNet dataset.
Korea Advanced Institute of Science and Technology（韩国科学技术院）
In this paper, we attack a few-shot open-set recognition (FSOSR) problem, which is a combination of few-shot learning (FSL) and open-set recognition (OSR). It aims to quickly adapt a model to a given small set of labeled samples while rejecting unseen class samples. Since OSR requires rich data and FSL considers closed-set classification, existing OSR and FSL methods show poor performances in solving FSOSR problems. The previous FSOSR method follows the pseudo-unseen class sample-based methods, which collect pseudo-unseen samples from the other dataset or synthesize samples to model unseen cla…
ConsDreamer: Advancing Multi-View Consistency for Zero-Shot Text-to-3D Generation
School of Artificial Intelligence, Nanjing University of Information Science and Technology（南京信息工程大学人工智能学院）；Department of Automation, Tsinghua University（清华大学自动化系）；Lenovo（联想）
TsCA: On the Semantic Consistency Alignment via Conditional Transport for Compositional Zero-Shot Learning
The Hong Kong Polytechnic University（香港理工大学）；Hong Kong Baptist University（香港浸会大学）；Shenzhen University（深圳大学）；Jilin University（吉林大学）；Hong Kong University of Science and Technology（香港科技大学）
SRI International（SRI国际）；Georgia Institute of Technology（佐治亚理工学院）
While models for Visual Question Answering (VQA) have steadily improved over the years, interacting with one quickly reveals that these models lack consistency. For instance, if a model answers "red" to "What color is the balloon?", it might answer "no" if asked, "Is the balloon red?". These responses violate simple notions of entailment and raise questions about how effectively VQA models ground language. In this work, we introduce a dataset, ConVQA, and metrics that enable quantitative evaluation of consistency in VQA. For a given observable fact in an image (e.g. the balloon's color), we ge…
NTT Corporation（日本电报电话公司）；NTT Human Informatics Laboratories（NTT人类信息学实验室）
Humans can obtain the knowledge of novel visual concepts from language descriptions, and we thus use the few-shot image classification task to investigate whether a machine learning model can have this capability. Our proposed model, LIDE (Learning from Image and DEscription), has a text decoder to generate the descriptions and a text encoder to obtain the text representations of machine- or user-generated descriptions. We confirmed that LIDE with machine-generated descriptions outperformed baseline models. Moreover, the performance was improved further with high-quality user-generated descrip…
Conventional training of deep neural networks requires a large number of the annotated image which is a laborious and time-consuming task, particularly for rare objects. Few-shot object detection (FSOD) methods offer a remedy by realizing robust object detection using only a few training samples per class. An unexplored challenge for FSOD is that instances from unlabeled novel classes that do not belong to the fixed set of training classes appear in the background. These objects behave similarly to label noise, leading to FSOD performance degradation. We develop a semi-supervised algorithm to …
Adaptive Semantic Consistency for Cross-domain Few-shot Classification
School of Artificial Intelligence and Automation, Huazhong University of Science and Technology（华中科技大学人工智能与自动化学院）；School of Electronic Information and Communications, Huazhong University of Science and Technology（华中科技大学电子信息与通信学院）
In this paper, we propose an approach to improve few-shot classification performance using a composite rotation based auxiliary task. Few-shot classification methods aim to produce neural networks that perform well for classes with a large number of training samples and classes with less number of training samples. They employ techniques to enable the network to produce highly discriminative features that are also very generic. Generally, the better the quality and generic-nature of the features produced by the network, the better is the performance of the network on few-shot learning. Our app…
ETH Zurich（苏黎世联邦理工学院）；Paul Scherrer Institute（保罗·谢勒研究所）
This work introduces and characterizes a fast parameterless filter based on the Helgason-Ludwig consistency conditions, used to improve the accuracy of analytical reconstructions of tomographic undersampled datasets. The filter, acting in the Radon domain, extrapolates intermediate projections between those existing. The resulting sinogram, doubled in views, is then reconstructed by a standard analytical method. Experiments with simulated data prove that the peak-signal-to-noise ratio of the results computed by filtered backprojection is improved up to 5-6 dB, if the filter is used prior to re…
Few-shot Defect Image Generation based on Consistency Modeling
Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；School of Artificial Intelligence, University of Chinese Academy of Sciences（中国科学院大学人工智能学院）；CASI Vision Technology CO., LTD.（中科视景科技有限公司）
It is a recognized fact that the classification accuracy of unseen classes in the setting of Generalized Zero-Shot Learning (GZSL) is much lower than that of traditional Zero-Shot Leaning (ZSL). One of the reasons is that an instance is always misclassified to the wrong domain. Here we refer to the seen and unseen classes as two domains respectively. We propose a new approach to distinguish whether the instances come from the seen or unseen classes. First the visual feature of instance is projected into the semantic space. Then the absolute norm difference between the projected semantic vector…
Key Laboratory of Multimedia Trusted Perception and Efficient Computing, Ministry of Education of China, School of Informatics, Xiamen University（中国教育部多媒体可信感知与高效计算重点实验室，厦门大学信息学院）；Fujian Key Laboratory of Pattern Recognition and Image Understanding, Xiamen University of Technology（厦门理工学院福建省模式识别与图像理解重点实验室）；School of Computer and Information Engineering, Xiamen University of Technology（厦门理工学院计算机与信息工程学院）
In this paper, we focus on improving the online face liveness detection system to enhance the security of the downstream face recognition system. Most of the existing frame-based methods are suffering from the prediction inconsistency across time. To address the issue, a simple yet effective solution based on temporal consistency is proposed. Specifically, in the training stage, to integrate the temporal consistency constraint, a temporal self-supervision loss and a class consistency loss are proposed in addition to the softmax cross-entropy loss. In the deployment stage, a training-free non-p…
