Real-world data is predominantly unbalanced and long-tailed, but deep models struggle to recognize rare classes in the presence of frequent classes. Often, classes can be accompanied by side information like textual descriptions, but it is not fully clear how to use them for learning with unbalanced long-tail data. Such descriptions have been mostly used in (Generalized) Zero-shot learning (ZSL), suggesting that ZSL with class descriptions may also be useful for long-tail distributions. We describe DRAGON, a late-fusion architecture for long-tail learning with class descriptors. It learns to (1) correct the bias towards head classes on a sample-by-sample basis; and (2) fuse information from class-descriptions to improve the tail-class accuracy. We also introduce new benchmarks CUB-LT, SUN-LT, AWA-LT for long-tail learning with class-descriptions, building on existing learning-with-attributes datasets and a version of Imagenet-LT with class descriptors. DRAGON outperforms state-of-the-art models on the new benchmark. It is also a new SoTA on existing benchmarks for GFSL with class descriptors (GFSL-d) and standard (vision-only) long-tailed learning ImageNet-LT, CIFAR-10, 100, and Places365.
Hefei University of Technology（合肥工业大学）；Hefei Comprehensive National Science Center（合肥综合性国家科学中心）；Zhengzhou University（郑州大学）；Singapore Management University（新加坡管理大学）
Few-shot classification studies the problem of quickly adapting a deep learner to understanding novel classes based on few support images. In this context, recent research efforts have been aimed at designing more and more complex classifiers that measure similarities between query and support images, but left the importance of feature embeddings seldom explored. We show that the reliance on sophisticated classifiers is not necessary, and a simple classifier applied directly to improved feature embeddings can instead outperform most of the leading methods in the literature. To this end, we pre…
Zero-shot learning (ZSL) for image classification focuses on recognizing novel categories that have no labeled data available for training. The learning is generally carried out with the help of mid-level semantic descriptors associated with each class. This semantic-descriptor space is generally shared by both seen and unseen categories. However, ZSL suffers from hubness, domain discrepancy and biased-ness towards seen classes. To tackle these problems, we propose a three-step approach to zero-shot learning. Firstly, a mapping is learned from the semantic-descriptor space to the image-feature…
The purpose of this paper is to describe one-shot-learning gesture recognition systems developed on the \textit{ChaLearn Gesture Dataset}. We use RGB and depth images and combine appearance (Histograms of Oriented Gradients) and motion descriptors (Histogram of Optical Flow) for parallel temporal segmentation and recognition. The Quadratic-Chi distance family is used to measure differences between histograms to capture cross-bin relationships. We also propose a new algorithm for trimming videos --- to remove all the unimportant frames from videos. We present two methods that use combination of…
University of Science and Technology of China（中国科学技术大学）
Few-shot segmentation aims at assigning a category label to each image pixel with few annotated samples. It is a challenging task since the dense prediction can only be achieved under the guidance of latent features defined by sparse annotations. Existing meta-learning method tends to fail in generating category-specifically discriminative descriptor when the visual features extracted from support images are marginalized in embedding space. To address this issue, this paper presents an adaptive tuning framework, in which the distribution of latent features across different episodes is dynamica…
Mangalore University（芒格洛尔大学）；KVG College of Engineering（KVG工程学院）
In this paper, we propose a novel approach for detecting the text present in videos and scene images based on the Multiscale Weber's Local Descriptor (MWLD). Given an input video, the shots are identified and the key frames are extracted based on their spatio-temporal relationship. From each key frame, we detect the local region information using WLD with different radius and neighborhood relationship of pixel values and hence obtained intensity enhanced key frames at multiple scales. These multiscale WLD key frames are merged together and then the horizontal gradients are computed using morph…
Nanjing Normal University（南京师范大学）；University of Wollongong（伍伦贡大学）；University of Sydney（悉尼大学）
This paper investigates a valuable setting called few-shot unsupervised domain adaptation (FS-UDA), which has not been sufficiently studied in the literature. In this setting, the source domain data are labelled, but with few-shot per category, while the target domain data are unlabelled. To address the FS-UDA setting, we develop a general UDA model to solve the following two key issues: the few-shot labeled data per category and the domain adaptation between support and query sets. Our model is general in that once trained it will be able to be applied to various FS-UDA tasks from the same so…
University of Campinas（坎皮纳斯大学）；Innopolis University（因诺波利斯大学）
Anomaly detection suffers from unbalanced data since anomalies are quite rare. Synthetically generated anomalies are a solution to such ill or not fully defined data. However, synthesis requires an expressive representation to guarantee the quality of the generated data. In this paper, we propose a two-level hierarchical latent space representation that distills inliers' feature-descriptors (through autoencoders) into more robust representations based on a variational family of distributions (through a variational autoencoder) for zero-shot anomaly generation. From the learned latent distribut…
CXR-LT 2024: A MICCAI challenge on long-tailed, multi-label, and zero-shot disease classification from chest X-ray
University of Minnesota（明尼苏达大学）；The University of Texas at Austin（德克萨斯大学奥斯汀分校）；Weill Cornell Medicine（威尔康奈尔医学院）；Mayo Clinic（梅奥诊所）；Shanghai Jiao Tong University（上海交通大学）；Shanghai AI Laboratory（上海人工智能实验室）；Yale University（耶鲁大学）；University of Pennsylvania（宾夕法尼亚大学）；Arizona State University（亚利桑那州立大学）；The University of Tokyo Hospital（东京大学医院）；Carnegie Mellon University（卡内基梅隆大学）；Millennium Institute for Intelligent Healthcare Engineering (iHEALTH)（千禧年智能医疗工程研究所）；National Center for Artificial Intelligence (CENIA)（国家人工智能中心）；Xiamen University（厦门大学）；Seoul National University（首尔大学）；Rensselaer Polytechnic Institute（伦斯勒理工学院）；Institute of High Performance Computing, A*STAR（A*STAR高性能计算研究院）；Thomas Jefferson University Hospital（托马斯杰斐逊大学医院）；Massachusetts Institute of Technology（麻省理工学院）；Beth Israel Deaconess Medical Center（贝斯以色列女执事医疗中心）；Harvard T.H. Chan School of Public Health（哈佛大学陈曾熙公共卫生学院）；National Institutes of Health（美国国立卫生研究院）；National Library of Medicine（美国国家医学图书馆）
本文介绍 MICCAI 挑战赛 CXR-LT 2024，扩展大规模长尾胸部 X 光数据集并设置三项分类任务，汇总多模态、生成式与零样本方法以提升临床诊断泛化能力。
ARC Centre of Excellence in Exciton Science（ARC激子科学卓越中心）；RMIT University（皇家墨尔本理工大学）；Institute for Frontier Materials, Deakin University（迪肯大学前沿材料研究所）
Establishing a data-driven pipeline for the discovery of novel materials requires the engineering of material features that can be feasibly calculated and can be applied to predict a material's target properties. Here we propose a new class of descriptors for describing crystal structures, which we term Robust One-Shot Ab initio (ROSA) descriptors. ROSA is computationally cheap and is shown to accurately predict a range of material properties. These simple and intuitive class of descriptors are generated from the energetics of a material at a low level of theory using an incomplete ab initio c…
State Key Lab of CAD&CG, Zhejiang University（浙江大学计算机辅助设计与图形学国家重点实验室）；Zhejiang Lab（之江实验室）
Few-shot learning (FSL) aims to classify images under low-data regimes, where the conventional pooled global feature is likely to lose useful local characteristics. Recent work has achieved promising performances by using deep descriptors. They generally take all deep descriptors from neural networks into consideration while ignoring that some of them are useless in classification due to their limited receptive field, e.g., task-irrelevant descriptors could be misleading and multiple aggregative descriptors from background clutter could even overwhelm the object's presence. In this paper, we a…
Navigating Data Scarcity using Foundation Models: A Benchmark of Few-Shot and Zero-Shot Learning Approaches in Medical Imaging
Cluster of Excellence “Machine Learning”（卓越集群“机器学习”）；University of Tübingen（蒂宾根大学）；Faculty of Health Sciences and Medicine, University of Lucerne（卢塞恩大学健康科学与医学院）
NTT Secure Platform Laboratories（NTT安全平台实验室）；NTT Communication Science Laboratories（NTT通信科学实验室）
We propose a method to infer domain-specific models such as classifiers for unseen domains, from which no data are given in the training phase, without domain semantic descriptors. When training and test distributions are different, standard supervised learning methods perform poorly. Zero-shot domain adaptation attempts to alleviate this problem by inferring models that generalize well to unseen domains by using training data in multiple source domains. Existing methods use observed semantic descriptors characterizing domains such as time information to infer the domain-specific models for th…
Zero-Shot Relation Extraction (ZRE) is the task of Relation Extraction where the training and test sets have no shared relation types. This very challenging domain is a good test of a model's ability to generalize. Previous approaches to ZRE reframed relation extraction as Question Answering (QA), allowing for the use of pre-trained QA models. However, this method required manually creating gold question templates for each new relation. Here, we do away with these gold templates and instead learn a model that can generate questions for unseen relations. Our technique can successfully translate…
AutoCLIP: Auto-tuning Zero-Shot Classifiers for Vision-Language Models
Bosch Center for Artificial Intelligence, Robert Bosch GmbH（博世人工智能中心，罗伯特·博世公司）；Bosch Center for Artificial Intelligence, Robert Bosch LLC（博世人工智能中心，罗伯特·博世有限责任公司）
D$^3$Fields: Dynamic 3D Descriptor Fields for Zero-Shot Generalizable Rearrangement
Boston Dynamics AI Institute（波士顿动力人工智能研究所）；University of Illinois, Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）；Stanford University（斯坦福大学）；Columbia University（哥伦比亚大学）；National University of Singapore（新加坡国立大学）
Knowledge transfer between tasks can improve the performance of learned models, but requires an accurate estimate of the inter-task relationships to identify the relevant knowledge to transfer. These inter-task relationships are typically estimated based on training data for each task, which is inefficient in lifelong learning settings where the goal is to learn each consecutive task rapidly from as little data as possible. To reduce this burden, we develop a lifelong learning method based on coupled dictionary learning that utilizes high-level task descriptions to model the inter-task relatio…
Multi-level Cross-modal Feature Alignment via Contrastive Learning towards Zero-shot Classification of Remote Sensing Image Scenes
Henan Key Laboratory of Big Data Analysis and Processing（河南省大数据分析与处理重点实验室）；Henan Engineering Laboratory of Spatial Information Processing（河南省空间信息处理工程实验室）；Henan Industrial Technology Academy of Spatio-Temporal Big Data, Henan University（河南大学河南时空大数据产业技术研究院）
Image-free Classifier Injection for Zero-Shot Classification
University of Tübingen（图宾根大学）；Technical University of Denmark（丹麦技术大学）；University of Trento（特伦托大学）；University of Copenhagen（哥本哈根大学）；Copenhagen University Hospital（哥本哈根大学医院）；MPI for Intelligent Systems（马克斯·普朗克智能系统研究所）
Nanjing University（南京大学）；University of Southern California（南加州大学）
Object recognition in the real-world requires handling long-tailed or even open-ended data. An ideal visual system needs to recognize the populated head visual concepts reliably and meanwhile efficiently learn about emerging new tail categories with a few training instances. Class-balanced many-shot learning and few-shot learning tackle one side of this problem, by either learning strong classifiers for head or learning to learn few-shot classifiers for the tail. In this paper, we investigate the problem of generalized few-shot learning (GFSL) -- a model during the deployment is required to le…
University of Würzburg（维尔茨堡大学）；Institute of Computer Science, University of Würzburg（维尔茨堡大学计算机科学研究所）
It is easier to hear birds than see them. However, they still play an essential role in nature and are excellent indicators of deteriorating environmental quality and pollution. Recent advances in Deep Neural Networks allow us to process audio data to detect and classify birds. This technology can assist researchers in monitoring bird populations and biodiversity. We propose a sound detection and classification pipeline to analyze complex soundscape recordings and identify birdcalls in the background. Our method learns from weak labels and few data and acoustically recognizes the bird species.…
University of Oxford（牛津大学）；Nanjing University of Science and Technology（南京理工大学）
Semantic-descriptor-based Generalized Zero-Shot Learning (GZSL) poses challenges in recognizing novel classes in the test phase. The development of generative models enables current GZSL techniques to probe further into the semantic-visual link, culminating in a two-stage form that includes a generator and a classifier. However, existing generation-based methods focus on enhancing the generator's effect while neglecting the improvement of the classifier. In this paper, we first analyze of two properties of the generated pseudo unseen samples: bias and homogeneity. Then, we perform variational …
Beijing Institute of Technology（北京理工大学）；Meituan（美团）
Prompt-tuning has shown appealing performance in few-shot classification by virtue of its capability in effectively exploiting pre-trained knowledge. This motivates us to check the hypothesis that prompt-tuning is also a promising choice for long-tailed classification, since the tail classes are intuitively few-shot ones. To achieve this aim, we conduct empirical studies to examine the hypothesis. The results demonstrate that prompt-tuning makes pretrained language models at least good long-tailed learners. For intuitions on why prompt-tuning can achieve good performance in long-tailed classif…
One Image is All You Need: Agentic One-Shot Image Generation via Text-Based World Models for Long-Tail Spatial Perception
Tsinghua University（清华大学）；SenseTime Research（商汤科技研究院）；Sun Yat-Sen University（中山大学）；Technical University of Munich（慕尼黑工业大学）；Heilbronn Data Science Center（海尔布隆数据科学中心）；Munich Data Institute（慕尼黑数据研究所）
Vision-language models (VLMs) such as CLIP have shown promising performance on a variety of recognition tasks using the standard zero-shot classification procedure -- computing similarity between the query image and the embedded words for each category. By only using the category name, they neglect to make use of the rich context of additional information that language affords. The procedure gives no intermediate understanding of why a category is chosen, and furthermore provides no mechanism for adjusting the criteria used towards this decision. We present an alternative framework for classif…
University of Augsburg（奥格斯堡大学）；TU Munich（慕尼黑工业大学）；Imperial College London（帝国理工学院）
Detecting hate speech, especially in low-resource languages, is a non-trivial challenge. To tackle this, we developed a tailored architecture based on frozen, pre-trained Transformers to examine cross-lingual zero-shot and few-shot learning, in addition to uni-lingual learning, on the HatEval challenge data set. With our novel attention-based classification block AXEL, we demonstrate highly competitive results on the English and Spanish subsets. We also re-sample the English subset, enabling additional, meaningful comparisons in the future.
Multi-label text classification (MLTC) aims to annotate documents with the most relevant labels from a number of candidate labels. In real applications, the distribution of label frequency often exhibits a long tail, i.e., a few labels are associated with a large number of documents (a.k.a. head labels), while a large fraction of labels are associated with a small number of documents (a.k.a. tail labels). To address the challenge of insufficient training data on tail label classification, we propose a Head-to-Tail Network (HTTN) to transfer the meta-knowledge from the data-rich head labels to …
