Trained on large datasets, deep learning (DL) can accurately classify videos into hundreds of diverse classes. However, video data is expensive to annotate. Zero-shot learning (ZSL) proposes one solution to this problem. ZSL trains a model once, and generalizes to new tasks whose classes are not present in the training dataset. We propose the first end-to-end algorithm for ZSL in video classification. Our training procedure builds on insights from recent video classification literature and uses a trainable 3D CNN to learn the visual features. This is in contrast to previous video ZSL methods, which use pretrained feature extractors. We also extend the current benchmarking paradigm: Previous techniques aim to make the test task unknown at training time but fall short of this goal. We encourage domain shift across training and test data and disallow tailoring a ZSL model to a specific test dataset. We outperform the state-of-the-art by a wide margin. Our code, evaluation procedure and model weights are available at github.com/bbrattoli/ZeroShotVideoClassification.
National University of Defense Technology（国防科技大学）；University of Oulu（奥卢大学）；Nanyang Technological University（南洋理工大学）；Stanford University（斯坦福大学）
Few-Shot Class-Incremental Learning (FSCIL) aims at incrementally learning novel classes from a few labeled samples by avoiding the overfitting and catastrophic forgetting simultaneously. The current protocol of FSCIL is built by mimicking the general class-incremental learning setting, while it is not totally appropriate due to the different data configuration, i.e., novel classes are all in the limited data regime. In this paper, we rethink the configuration of FSCIL with the open-set hypothesis by reserving the possibility in the first session for incoming categories. To assign better perfo…
University of Chinese Academy of Sciences（中国科学院大学）；Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；CAS Center for Excellence in Brain Science and Intelligence Technology（中国科学院脑科学与智能技术卓越创新中心）
Pedestrian attribute recognition aims to assign multiple attributes to one pedestrian image captured by a video surveillance camera. Although numerous methods are proposed and make tremendous progress, we argue that it is time to step back and analyze the status quo of the area. We review and rethink the recent progress from three perspectives. First, given that there is no explicit and complete definition of pedestrian attribute recognition, we formally define and distinguish pedestrian attribute recognition from other similar tasks. Second, based on the proposed definition, we expose the lim…
University of Chinese Academy of Sciences（中国科学院大学）；Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；CAS Center for Excellence in Brain Science and Intelligence Technology（中国科学院脑科学与智能技术卓越创新中心）
Despite various methods are proposed to make progress in pedestrian attribute recognition, a crucial problem on existing datasets is often neglected, namely, a large number of identical pedestrian identities in train and test set, which is not consistent with practical application. Thus, images of the same pedestrian identity in train set and test set are extremely similar, leading to overestimated performance of state-of-the-art methods on existing datasets. To address this problem, we propose two realistic datasets PETA\textsubscript{$zs$} and RAPv2\textsubscript{$zs$} following zero-shot se…
Rethinking Plant Disease Diagnosis: Bridging the Academic-Practical Gap with Vision Transformers and Zero-Shot Learning
Department of Computer Science, Mohamed El Bachir El Ibrahimi University（计算机科学系，莫亨德·埃巴希里·埃布拉希米大学）；Laboratory of Physical Chemistry and Biological Materials（物理化学与生物材料实验室）；Laboratory of Informatics and its Applications of M’sila, Mohamed Boudiaf University（信息学及其应用实验室，莫赫梅德·布迪亚夫大学）；Department of Computer Science, Mohamed Boudiaf University（计算机科学系，莫赫梅德·布迪亚夫大学）；Department of agricultural science, Mohamed El Bachir El Ibrahimi University（农业科学系，莫亨德·埃巴希里·埃布拉希米大学）；Laboratoire d’Amélioration et de Développement de la Production Végétale et Animale (LADPVA), University of Ferhat ABBAS (UFAS-Sétif1)（植物和动物生产改良与发展实验室（LADPVA），费哈特·阿巴斯大学（UFAS-Sétif1））
Continual Alignment for SAM: Rethinking Foundation Models for Medical Image Segmentation in Continual Learning
Xi’an Jiaotong University（西安交通大学）；School of Information and Communications Engineering, Xi’an Jiaotong University（信息与通信工程学院）；School of Mathematics and Statistics, Xi’an Jiaotong University（数学与统计学学院）
One of the most critical problems in weight-sharing neural architecture search is the evaluation of candidate models within a predefined search space. In practice, a one-shot supernet is trained to serve as an evaluator. A faithful ranking certainly leads to more accurate searching results. However, current methods are prone to making misjudgments. In this paper, we prove that their biased evaluation is due to inherent unfairness in the supernet training. In view of this, we propose two levels of constraints: expectation fairness and strict fairness. Particularly, strict fairness ensures equal…
UC, Berkeley（加州大学伯克利分校）；MILA, Université de Montréal（蒙特利尔大学MILA）；DiDi（滴滴）；UC, San Diego（加州大学圣地亚哥分校）；CIFAR（加拿大高等研究院）
Domain adaptation (DA) is a technique that transfers predictive models trained on a labeled source domain to an unlabeled target domain, with the core difficulty of resolving distributional shift between domains. Currently, most popular DA algorithms are based on distributional matching (DM). However in practice, realistic domain shifts (RDS) may violate their basic assumptions and as a result these methods will fail. In this paper, in order to devise robust DA algorithms, we first systematically analyze the limitations of DM based methods, and then build new benchmarks with more realistic dom…
Czech Technical University in Prague（捷克布拉格技术大学）；NAVER LABS Europe（NAVER欧洲实验室）
Few-shot action recognition, i.e. recognizing new action classes given only a few examples, benefits from incorporating temporal information. Prior work either encodes such information in the representation itself and learns classifiers at test time, or obtains frame-level features and performs pairwise temporal matching. We first evaluate a number of matching-based approaches using features from spatio-temporal backbones, a comparison missing from the literature, and show that the gap in performance between simple baselines and more complicated methods is significantly reduced. Inspired by th…
Inception Institute of Artificial Intelligence（初始人工智能研究院）；University of Macau（澳门大学）；Terminus Group（特斯联集团）
The pioneering method for unsupervised meta-learning, CACTUs, is a clustering-based approach with pseudo-labeling. This approach is model-agnostic and can be combined with supervised algorithms to learn from unlabeled data. However, it often suffers from label inconsistency or limited diversity, which leads to poor performance. In this work, we prove that the core reason for this is lack of a clustering-friendly property in the embedding space. We address this by minimizing the inter- to intra-class similarity ratio to provide clustering-friendly embedding features, and validate our approach t…
Rethinking Position Embedding as a Context Controller for Multi-Reference and Multi-Shot Video Generation
Wuhan University（武汉大学）；University of Science and Technology of China（中国科学技术大学）；Hong Kong University of Science and Technology（香港科技大学）；Tsinghua University（清华大学）
Online few-shot learning describes a setting where models are trained and evaluated on a stream of data while learning emerging classes. While prior work in this setting has achieved very promising performance on instance classification when learning from data-streams composed of a single indoor environment, we propose to extend this setting to consider object classification on a series of several indoor environments, which is likely to occur in applications such as robotics. Importantly, our setting, which we refer to as online few-shot continual learning, injects the well-studied issue of ca…
Monash University（莫纳什大学）；The University of Melbourne（墨尔本大学）
Single image-level annotations only correctly describe an often small subset of an image's content, particularly when complex real-world scenes are depicted. While this might be acceptable in many classification scenarios, it poses a significant challenge for applications where the set of classes differs significantly between training and test time. In this paper, we take a closer look at the implications in the context of $\textit{few-shot learning}$. Splitting the input samples into patches and encoding these via the help of Vision Transformers allows us to establish semantic correspondences…
Most existing works on few-shot object detection (FSOD) focus on a setting where both pre-training and few-shot learning datasets are from a similar domain. However, few-shot algorithms are important in multiple domains; hence evaluation needs to reflect the broad applications. We propose a Multi-dOmain Few-Shot Object Detection (MoFSOD) benchmark consisting of 10 datasets from a wide range of domains to evaluate FSOD algorithms. We comprehensively analyze the impacts of freezing layers, different architectures, and different pre-training datasets on FSOD performance. Our empirical results sho…
Australian National University（澳大利亚国立大学）；Data61/CSIRO（联邦科学与工业研究组织Data61）；University of Oxford（牛津大学）；National University of Defense Technology（国防科技大学）
The majority of existing few-shot learning methods describe image relations with binary labels. However, such binary relations are insufficient to teach the network complicated real-world relations, due to the lack of decision smoothness. Furthermore, current few-shot learning models capture only the similarity via relation labels, but they are not exposed to class concepts associated with objects, which is likely detrimental to the classification performance due to underutilization of the available class labels. To paraphrase, children learn the concept of tiger from a few of actual examples …
Tencent Youtu Lab（腾讯优图实验室）；CATL（宁德时代）；East China Normal University（华东师范大学）
Few-shot learning problem focuses on recognizing unseen classes given a few labeled images. In recent effort, more attention is paid to fine-grained feature embedding, ignoring the relationship among different distance metrics. In this paper, for the first time, we investigate the contributions of different distance metrics, and propose an adaptive fusion scheme, bringing significant improvements in few-shot classification. We start from a naive baseline of confidence summation and demonstrate the necessity of exploiting the complementary property of different distance metrics. By finding the …
MICM: Rethinking Unsupervised Pretraining for Enhanced Few-shot Learning
School of Computer Science and Technology, Huazhong University of Science and Technology（华中科技大学计算机科学与技术学院）；School of Computer Science, Peking University（北京大学计算机学院）
Google Research（谷歌研究院）；MIT CSAIL（麻省理工学院计算机科学与人工智能实验室）
The focus of recent meta-learning research has been on the development of learning algorithms that can quickly adapt to test time tasks with limited data and low computational cost. Few-shot learning is widely used as one of the standard benchmarks in meta-learning. In this work, we show that a simple baseline: learning a supervised or self-supervised representation on the meta-training set, followed by training a linear classifier on top of this representation, outperforms state-of-the-art few-shot learning methods. An additional boost can be achieved through the use of self-distillation. Thi…
Robust Model-Agnostic Meta-Learning (MAML) is usually adopted to train a meta-model which may fast adapt to novel classes with only a few exemplars and meanwhile remain robust to adversarial attacks. The conventional solution for robust MAML is to introduce robustness-promoting regularization during meta-training stage. With such a regularization, previous robust MAML methods simply follow the typical MAML practice that the number of training shots should match with the number of test shots to achieve an optimal adaptation performance. However, although the robustness can be largely improved, …
UiT The Arctic University of Norway（挪威北极大学）；Tsinghua University（清华大学）；Sun Yat-sen University（中山大学）；Massachusetts Institute of Technology（麻省理工学院）；Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；Carnegie Mellon University（卡内基梅隆大学）
Graph convolutional neural networks have recently shown great potential for the task of zero-shot learning. These models are highly sample efficient as related concepts in the graph structure share statistical strength allowing generalization to new classes when faced with a lack of data. However, multi-layer architectures, which are required to propagate knowledge to distant nodes in the graph, dilute the knowledge by performing extensive Laplacian smoothing at each layer and thereby consequently decrease performance. In order to still enjoy the benefit brought by the graph structure while pr…
Zero-shot learning (ZSL) aims to recognize instances of unseen classes solely based on the semantic descriptions of the classes. Existing algorithms usually formulate it as a semantic-visual correspondence problem, by learning mappings from one feature space to the other. Despite being reasonable, previous approaches essentially discard the highly precious discriminative power of visual features in an implicit way, and thus produce undesirable results. We instead reformulate ZSL as a conditioned visual classification problem, i.e., classifying visual features based on the classifiers learned f…
Few-shot recognition involves training an image classifier to distinguish novel concepts at test time using few examples (shot). Existing approaches generally assume that the shot number at test time is known in advance. This is not realistic, and the performance of a popular and foundational method has been shown to suffer when train and test shots do not match. We conduct a systematic empirical study of this phenomenon. In line with prior work, we find that shot sensitivity is broadly present across metric-based few-shot learners, but in contrast to prior work, larger neural architectures pr…
DroneSR: Rethinking Few-shot Thermal Image Super-Resolution from Drone-based Perspective
School of Information Engineering, Guangdong University of Technology（广东工业大学信息工程学院）；Guangdong Power Grid, Ltd.（广东电网有限公司）；School of Computer Science, Sun Yat-sen University（中山大学计算机学院）
A common problem with most zero and few-shot learning approaches is they suffer from bias towards seen classes resulting in sub-optimal performance. Existing efforts aim to utilize unlabeled images from unseen classes (i.e transductive zero-shot) during training to enable generalization. However, this limits their use in practical scenarios where data from target unseen classes is unavailable or infeasible to collect. In this work, we present a practical setting of inductive zero and few-shot learning, where unlabeled images from other out-of-data classes, that do not belong to seen or unseen …
University of Illinois at Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）
Despite achieving state-of-the-art zero-shot performance, existing vision-language models still fall short of few-shot transfer ability on domain-specific problems. Classical fine-tuning often fails to prevent highly expressive models from exploiting spurious correlations. Although model-agnostic meta-learning (MAML) presents as a natural alternative for few-shot transfer learning, the expensive computation due to implicit second-order optimization limits its use on large-scale vision-language models such as CLIP. While much literature has been devoted to exploring alternative optimization str…
Rethinking Few-shot 3D Point Cloud Semantic Segmentation
ETH Zurich（苏黎世联邦理工学院）；Pioneer Centre for Artificial Intelligence, University of Copenhagen（哥本哈根大学人工智能先锋中心）；Institute for Infocomm Research, A*STAR（新加坡科技研究局信息通信研究院）；CAIDAS & IFI, University of Wurzburg（维尔茨堡大学CAIDAS与IFI机构）
CausalFSFG: Rethinking Few-Shot Fine-Grained Visual Categorization from Causal Perspective
Wangxuan Institute of Computer Technology, Peking University（北京大学王轩计算机技术研究所）；School of Intelligence Science and Technology, University of Science and Technology Beijing（北京科技大学智能科学与技术学院）
Rethinking the Role of Feature Engineering and Learning Strategies in Few-Shot Hidden Emotion Recognition
Hefei University of Technology（合肥工业大学）；University College London（伦敦大学学院）；The University of Sydney（悉尼大学）；Beijing QBoson Quantum Technology Co., Ltd.（北京量子芯光科技有限公司）；Institute of Artificial Intelligence, Hefei Comprehensive National Science Center（合肥综合性国家科学中心人工智能研究院）；Beihang University（北京航空航天大学）；The University of New South Wales（新南威尔士大学）；United Arab Emirates University（阿拉伯联合酋长国大学）
The University of Queensland（昆士兰大学）；University of Electronic Science and Technology of China（电子科技大学）
Zero-shot learning (ZSL) is commonly used to address the very pervasive problem of predicting unseen classes in fine-grained image classification and other tasks. One family of solutions is to learn synthesised unseen visual samples produced by generative models from auxiliary semantic information, such as natural language descriptions. However, for most of these models, performance suffers from noise in the form of irrelevant image backgrounds. Further, most methods do not allocate a calculated weight to each semantic patch. Yet, in the real world, the discriminative power of features can be …
University of Electronic Science and Technology of China（电子科技大学）；Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；University of Chinese Academy of Sciences（中国科学院大学）；CAS Center for Excellence in Brain Science and Intelligence Technology（中国科学院脑科学与智能技术卓越创新中心）
Due to balanced accuracy and speed, one-shot models which jointly learn detection and identification embeddings, have drawn great attention in multi-object tracking (MOT). However, the inherent differences and relations between detection and re-identification (ReID) are unconsciously overlooked because of treating them as two isolated tasks in the one-shot tracking paradigm. This leads to inferior performance compared with existing two-stage methods. In this paper, we first dissect the reasoning process for these two tasks, which reveals that the competition between them inevitably would destr…
Rethinking the Sample Relations for Few-Shot Classification
School of Big Data and Software Engineering, Chongqing University（重庆大学大数据与软件学院）；Chongqing University（重庆大学）；Fowler College of Business, San Diego State University（圣地亚哥州立大学福勒商学院）；San Diego State University（圣地亚哥州立大学）
The Australian National University（澳大利亚国立大学）；Data61-CSIRO（澳大利亚联邦科学与工业研究组织Data61）；Mohamed Bin Zayed University of AI（穆罕默德·本·扎耶德人工智能大学）
Conditional GANs (cGAN), in their rudimentary form, suffer from critical drawbacks such as the lack of diversity in generated outputs and distortion between the latent and output manifolds. Although efforts have been made to improve results, they can suffer from unpleasant side-effects such as the topology mismatch between latent and output spaces. In contrast, we tackle this problem from a geometrical perspective and propose a novel training mechanism that increases both the diversity and the visual quality of a vanilla cGAN, by systematically encouraging a bi-lipschitz mapping between the la…
