In the context of few-shot learning, one cannot measure the generalization ability of a trained classifier using validation sets, due to the small number of labeled samples. In this paper, we are interested in finding alternatives to answer the question: is my classifier generalizing well to previously unseen data? We first analyze the reasons for the variability of generalization performances. We then investigate the case of using transfer-based solutions, and consider three settings: i) supervised where we only have access to a few labeled samples, ii) semi-supervised where we have access to both a few labeled samples and a set of unlabeled samples and iii) unsupervised where we only have access to unlabeled samples. For each setting, we propose reasonable measures that we empirically demonstrate to be correlated with the generalization ability of considered classifiers. We also show that these simple measures can be used to predict generalization up to a certain confidence. We conduct our experiments on standard few-shot vision datasets.
Nanjing University（南京大学）；University of Southern California（南加州大学）
Object recognition in the real-world requires handling long-tailed or even open-ended data. An ideal visual system needs to recognize the populated head visual concepts reliably and meanwhile efficiently learn about emerging new tail categories with a few training instances. Class-balanced many-shot learning and few-shot learning tackle one side of this problem, by either learning strong classifiers for head or learning to learn few-shot classifiers for the tail. In this paper, we investigate the problem of generalized few-shot learning (GFSL) -- a model during the deployment is required to le…
Nanjing University（南京大学）；Ohio State University（俄亥俄州立大学）
Few-shot learning (FSL) aims to generate a classifier using limited labeled examples. Many existing works take the meta-learning approach, constructing a few-shot learner that can learn from few-shot examples to generate a classifier. Typically, the few-shot learner is constructed or meta-trained by sampling multiple few-shot tasks in turn and optimizing the few-shot learner's performance in generating classifiers for those tasks. The performance is measured by how well the resulting classifiers classify the test (i.e., query) examples of those tasks. In this paper, we point out two potential …
The prototypical network is a prototype classifier based on meta-learning and is widely used for few-shot learning because it classifies unseen examples by constructing class-specific prototypes without adjusting hyper-parameters during meta-testing. Interestingly, recent research has attracted a lot of attention, showing that training a new linear classifier, which does not use a meta-learning algorithm, performs comparably with the prototypical network. However, the training of a new linear classifier requires the retraining of the classifier every time a new class appears. In this paper, we…
University of Maryland, Baltimore County（马里兰大学巴尔的摩县分校）；University of California, Davis（加利福尼亚大学戴维斯分校）
Few-shot image classification, where the goal is to generalize to tasks with limited labeled data, has seen great progress over the years. However, the classifiers are vulnerable to adversarial examples, posing a question regarding their generalization capabilities. Recent works have tried to combine meta-learning approaches with adversarial training to improve the robustness of few-shot classifiers. We show that a simple transfer-learning based approach can be used to train adversarially robust few-shot classifiers. We also present a method for novel classification task based on calibrating t…
Rice University（莱斯大学）；University of Texas, Austin（德克萨斯大学奥斯汀分校）
Recent papers have suggested that transfer learning can outperform sophisticated meta-learning methods for few-shot image classification. We take this hypothesis to its logical conclusion, and suggest the use of an ensemble of high-quality, pre-trained feature extractors for few-shot image classification. We show experimentally that a library of pre-trained feature extractors combined with a simple feed-forward network learned with an L2-regularizer can be an excellent option for solving cross-domain few-shot image classification. Our experimental results suggest that this simpler sample-effic…
In few-shot classification, the primary goal is to learn representations from a few samples that generalize well for novel classes. In this paper, we propose an efficient low displacement rank (LDR) regularization strategy termed Ortho-Shot; a technique that imposes orthogonal regularization on the convolutional layers of a few-shot classifier, which is based on the doubly-block toeplitz (DBT) matrix structure. The regularized convolutional layers of the few-shot classifier enhances model generalization and intra-class feature embeddings that are crucial for few-shot learning. Overfitting is a…
National University of Ireland, Galway（爱尔兰国立大学戈尔韦分校）
Machine Learning (ML) techniques for image classification routinely require many labelled images for training the model and while testing, we ought to use images belonging to the same domain as those used for training. In this paper, we overcome the two main hurdles of ML, i.e. scarcity of data and constrained prediction of the classification model. We do this by introducing a visual classifier which uses a concept of transfer learning, namely Zero-Shot Learning (ZSL), and standard Natural Language Processing techniques. We train a classifier by mapping labelled images to their textual descrip…
The estimation of the generalization error of classifiers often relies on a validation set. Such a set is hardly available in few-shot learning scenarios, a highly disregarded shortcoming in the field. In these scenarios, it is common to rely on features extracted from pre-trained neural networks combined with distance-based classifiers such as nearest class mean. In this work, we introduce a Gaussian model of the feature distribution. By estimating the parameters of this model, we are able to predict the generalization error on new classification tasks with few samples. We observe that accura…
University of Technology Sydney（悉尼科技大学）；University of Washington（华盛顿大学）
We study many-class few-shot (MCFS) problem in both supervised learning and meta-learning settings. Compared to the well-studied many-class many-shot and few-class few-shot problems, the MCFS problem commonly occurs in practical applications but has been rarely studied in previous literature. It brings new challenges of distinguishing between many classes given only a few training samples per class. In this paper, we leverage the class hierarchy as a prior knowledge to train a coarse-to-fine classifier that can produce accurate predictions for MCFS problem in both settings. The propose model, …
Mila（米拉研究所）；Université de Montréal（蒙特利尔大学）；École Polytechnique de Montréal（蒙特利尔理工学院）
Empirical science of neural scaling laws is a rapidly growing area of significant importance to the future of machine learning, particularly in the light of recent breakthroughs achieved by large-scale pre-trained models such as GPT-3, CLIP and DALL-e. Accurately predicting the neural network performance with increasing resources such as data, compute and model size provides a more comprehensive evaluation of different approaches across multiple scales, as opposed to traditional point-wise comparisons of fixed-size models on fixed-size benchmarks, and, most importantly, allows for focus on the…
IPEC: Test-Time Incremental Prototype Enhancement Classifier for Few-Shot Learning
College of Intelligent Robotics and Advanced Manufacturing, Fudan University（智能机器人与先进制造学院，复旦大学）；School of Microelectronics, Fudan University（微电子学院，复旦大学）；Key Laboratory of Advanced Control and Optimization for Chemical Processes of Ministry of Education, East China University of Science and Technology（教育部先进化工过程控制与优化重点实验室，东华大学）
In few-shot domain adaptation (FDA), classifiers for the target domain are trained with accessible labeled data in the source domain (SD) and few labeled data in the target domain (TD). However, data usually contain private information in the current era, e.g., data distributed on personal phones. Thus, the private information will be leaked if we directly access data in SD to train a target-domain classifier (required by FDA methods). In this paper, to thoroughly prevent the privacy leakage in SD, we consider a very challenging problem setting, where the classifier for the TD has to be traine…
Federal Technology University（联邦技术大学）；University of São Paulo（圣保罗大学）；Universidade Estadual Paulista（保利斯塔州立大学）；Eindhoven University of Technology（埃因霍温理工大学）
For many machine learning algorithms, predictive performance is critically affected by the hyperparameter values used to train them. However, tuning these hyperparameters can come at a high computational cost, especially on larger datasets, while the tuned settings do not always significantly outperform the default values. This paper proposes a recommender system based on meta-learning to identify exactly when it is better to use default values and when to tune hyperparameters for each new dataset. Besides, an in-depth analysis is performed to understand what they take into account for their d…
ETH AI Center, ETH Zürich（苏黎世联邦理工学院ETH人工智能中心）；Institute of Neuroinformatics, University of Zürich and ETH Zürich（苏黎世大学与苏黎世联邦理工学院神经信息学研究所）；Soft Robotics Lab, ETH Zürich（苏黎世联邦理工学院软机器人实验室）；Dept. of Computer Science, ETH Zürich（苏黎世联邦理工学院计算机科学系）
We introduce meta-learning algorithms that perform zero-shot weight-space adaptation of neural network models to unseen tasks. Our methods repurpose the popular generative image synthesis techniques of natural language guidance and diffusion models to generate neural network weights adapted for tasks. We first train an unconditional generative hypernetwork model to produce neural network weights; then we train a second "guidance" model that, given a natural language task description, traverses the hypernetwork latent space to find high-performance task-adapted weights in a zero-shot manner. We…
University of Edinburgh（爱丁堡大学）；Samsung AI Centre（三星人工智能中心）；University of Chinese Academy of Sciences（中国科学院大学）
Current state-of-the-art few-shot learners focus on developing effective training procedures for feature representations, before using simple, e.g. nearest centroid, classifiers. In this paper, we take an orthogonal approach that is agnostic to the features used and focus exclusively on meta-learning the actual classifier layer. Specifically, we introduce MetaQDA, a Bayesian meta-learning generalization of the classic quadratic discriminant analysis. This setup has several benefits of interest to practitioners: meta-learning is fast and memory-efficient, without the need to fine-tune features.…
Image-free Classifier Injection for Zero-Shot Classification
University of Tübingen（图宾根大学）；Technical University of Denmark（丹麦技术大学）；University of Trento（特伦托大学）；University of Copenhagen（哥本哈根大学）；Copenhagen University Hospital（哥本哈根大学医院）；MPI for Intelligent Systems（马克斯·普朗克智能系统研究所）
Peking University（北京大学）；University of Oxford（牛津大学）；JD Explore Academy（京东探索研究院）；Wuhan University（武汉大学）；Peng Cheng Laboratory（鹏城实验室）
Few-shot class-incremental learning (FSCIL) has been a challenging problem as only a few training samples are accessible for each novel class in the new sessions. Finetuning the backbone or adjusting the classifier prototypes trained in the prior sessions would inevitably cause a misalignment between the feature and classifier of old classes, which explains the well-known catastrophic forgetting problem. In this paper, we deal with this misalignment dilemma in FSCIL inspired by the recently discovered phenomenon named neural collapse, which reveals that the last-layer features of the same clas…
Saarland University（萨尔大学）；American International University-Bangladesh（孟加拉国美国国际大学）
Autism Spectrum Disorder (ASD) is on the rise and constantly growing. Earlier identify of ASD with the best outcome will allow someone to be safe and healthy by proper nursing. Humans can hardly estimate the present condition and stage of ASD by measuring primary symptoms. Therefore, it is being necessary to develop a method that will provide the best outcome and measurement of ASD. This paper aims to show several measurements that implemented in several classifiers. Among them, Support Vector Machine (SVM) provides the best result and under SVM, there are also some kernels to perform. Among t…
University of Oxford（牛津大学）；Nanjing University of Science and Technology（南京理工大学）
Semantic-descriptor-based Generalized Zero-Shot Learning (GZSL) poses challenges in recognizing novel classes in the test phase. The development of generative models enables current GZSL techniques to probe further into the semantic-visual link, culminating in a two-stage form that includes a generator and a classifier. However, existing generation-based methods focus on enhancing the generator's effect while neglecting the improvement of the classifier. In this paper, we first analyze of two properties of the generated pseudo unseen samples: bias and homogeneity. Then, we perform variational …
Warsaw University of Technology（华沙理工大学）；Systems Research Institute Polish Academy of Sciences（波兰科学院系统研究所）
One of important areas of machine learning research is zero-shot learning. It is applied when properly labeled training data set is not available. A number of zero-shot algorithms have been proposed and experimented with. However, none of them seems to be the "overall winner". In situations like this, it may be possible to develop a meta-classifier that would combine "best aspects" of individual classifiers and outperform all of them. In this context, the goal of this contribution is twofold. First, multiple state-of-the-art zero-shot learning methods are compared for standard benchmark datase…
Few-shot class-incremental learning (FSCIL) aims to design machine learning algorithms that can continually learn new concepts from a few data points, without forgetting knowledge of old classes. The difficulty lies in that limited data from new classes not only lead to significant overfitting issues but also exacerbate the notorious catastrophic forgetting problems. Moreover, as training data come in sequence in FSCIL, the learned classifier can only provide discriminative information in individual sessions, while FSCIL requires all classes to be involved for evaluation. In this paper, we add…
AutoCLIP: Auto-tuning Zero-Shot Classifiers for Vision-Language Models
Bosch Center for Artificial Intelligence, Robert Bosch GmbH（博世人工智能中心，罗伯特·博世公司）；Bosch Center for Artificial Intelligence, Robert Bosch LLC（博世人工智能中心，罗伯特·博世有限责任公司）
In this paper, we predict the likelihood of a player making a shot in basketball from multiagent trajectories. Previous approaches to similar problems center on hand-crafting features to capture domain specific knowledge. Although intuitive, recent work in deep learning has shown this approach is prone to missing important predictive features. To circumvent this issue, we present a convolutional neural network (CNN) approach where we initially represent the multiagent behavior as an image. To encode the adversarial nature of basketball, we use a multi-channel image which we then feed into a CN…
University of Illinois Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）
Learning accurate classifiers for novel categories from very few examples, known as few-shot image classification, is a challenging task in statistical machine learning and computer vision. The performance in few-shot classification suffers from the bias in the estimation of classifier parameters; however, an effective underlying bias reduction technique that could alleviate this issue in training few-shot classifiers has been overlooked. In this work, we demonstrate the effectiveness of Firth bias reduction in few-shot classification. Theoretically, Firth bias reduction removes the $O(N^{-1})…
Tsinghua University（清华大学）；University of Texas at Austin（德克萨斯大学奥斯汀分校）；Central South University（中南大学）；Kuaishou Technology Co., Ltd.（快手科技有限公司）；Harvard University（哈佛大学）
Tshwane University of Technology（茨瓦内理工大学）；Waterford Institute of Technology（沃特福德理工学院）
An ensemble of classifiers combines several single classifiers to deliver a final prediction or classification decision. An increasingly provoking question is whether such systems can outperform the single best classifier. If so, what form of an ensemble of classifiers (also known as multiple classifier learning systems or multiple classifiers) yields the most significant benefits in the size or diversity of the ensemble itself? Given that the tests used to detect autism traits are time-consuming and costly, developing a system that will provide the best outcome and measurement of autism spect…
Symanto Research（Symanto研究院）；Universitat Politècnica de València（瓦伦西亚理工大学）；PRHLT Research Center（PRHLT研究中心）
Zero-shot text classifiers based on label descriptions embed an input text and a set of labels into the same space: measures such as cosine similarity can then be used to select the most similar label description to the input text as the predicted label. In a true zero-shot setup, designing good label descriptions is challenging because no development set is available. Inspired by the literature on Learning with Disagreements, we look at how probabilistic models of repeated rating analysis can be used for selecting the best label descriptions in an unsupervised fashion. We evaluate our method …
One-shot learning is proposed to make a pretrained classifier workable on a new dataset based on one labeled samples from each pattern. However, few of researchers consider whether the dataset itself supports one-shot learning. In this paper, we propose a set of definitions to explain what kind of datasets can support one-shot learning and propose the concept "absolute generalization". Based on these definitions, we proposed a method to build an absolutely generalizable classifier. The proposed method concatenates two samples as a new single sample, and converts a classification problem to an …
ST Engineering-SUTD Cyber Security Laboratory（ST Engineering-新加坡科技设计大学网络安全实验室）；Singapore University of Technology and Design（新加坡科技设计大学）；Ben-Gurion University（本-古里安大学）；University of Oslo（奥斯陆大学）
Few-shot classifiers excel under limited training samples, making them useful in applications with sparsely user-provided labels. Their unique relative prediction setup offers opportunities for novel attacks, such as targeting support sets required to categorise unseen test samples, which are not available in other machine learning setups. In this work, we propose a detection strategy to identify adversarial support sets, aimed at destroying the understanding of a few-shot classifier for a certain class. We achieve this by introducing the concept of self-similarity of a support set and by empl…
University of Michigan（密歇根大学）；Michigan Engineering Services（密歇根工程服务公司）；US Army DEVCOM GVSC（美国陆军作战能力发展司令部地面车辆系统中心）
In search, exploration, and reconnaissance tasks performed with autonomous ground vehicles, an image classification capability is needed for specifically identifying targeted objects (relevant classes) and at the same time recognize when a candidate image does not belong to anyone of the relevant classes (irrelevant images). In this paper, we present an open-set low-shot classifier that uses, during its training, a modest number (less than 40) of labeled images for each relevant class, and unlabeled irrelevant images that are randomly selected at each epoch of the training process. The new cla…
Stanford University（斯坦福大学）；Carnegie Mellon University（卡内基梅隆大学）；Google（谷歌）；Argo AI
For machine learning models trained with limited labeled training data, validation stands to become the main bottleneck to reducing overall annotation costs. We propose a statistical validation algorithm that accurately estimates the F-score of binary classifiers for rare categories, where finding relevant examples to evaluate on is particularly challenging. Our key insight is that simultaneous calibration and importance sampling enables accurate estimates even in the low-sample regime (< 300 samples). Critically, we also derive an accurate single-trial estimator of the variance of our method …
Singapore University of Technology and Design（新加坡科技设计大学）；Ben-Gurion University of the Negev（内盖夫本-古里安大学）；University of Oslo（奥斯陆大学）
Few-shot classifiers have been shown to exhibit promising results in use cases where user-provided labels are scarce. These models are able to learn to predict novel classes simply by training on a non-overlapping set of classes. This can be largely attributed to the differences in their mechanisms as compared to conventional deep networks. However, this also offers new opportunities for novel attackers to induce integrity attacks against such models, which are not present in other machine learning setups. In this work, we aim to close this gap by studying a conceptually simple approach to def…
Generalized zero-shot learning (GZSL) is one of the most realistic but challenging problems due to the partiality of the classifier to supervised classes, especially under the class-inductive instance-inductive (CIII) training setting, where testing data are not available. Instance-borrowing methods and synthesizing methods solve it to some extent with the help of testing semantics, but therefore neither can be used under CIII. Besides, the latter require the training process of a classifier after generating examples. In contrast, a novel non-transductive regularization under CIII called Seman…
DPSeq: A Novel and Efficient Digital Pathology Classifier for Predicting Cancer Biomarkers using Sequencer Architecture
School of Data Science, University of Science and Technology of China（中国科学技术大学数据科学学院）；School of Management, University of Science and Technology of China（中国科学技术大学管理学院）；School of Population Health, UNSW Sydney（新南威尔士大学悉尼分校人口健康学院）；Genmab Inc.（杰特贝公司）
COOkeD: Ensemble-based OOD detection in the era of zero-shot CLIP
Aalborg University（奥尔堡大学）；ENSTA, Institut Polytechnique de Paris（法国国立高等先进技术学校，巴黎综合理工学院）；University of Barcelona（巴塞罗那大学）；Computer Vision Center（计算机视觉中心）
