We study the impact of using rich and diverse textual descriptions of classes for zero-shot learning (ZSL) on ImageNet. We create a new dataset ImageNet-Wiki that matches each ImageNet class to its corresponding Wikipedia article. We show that merely employing these Wikipedia articles as class descriptions yields much higher ZSL performance than prior works. Even a simple model using this type of auxiliary data outperforms state-of-the-art models that rely on standard features of word embedding encodings of class names. These results highlight the usefulness and importance of textual descriptions for ZSL, as well as the relative importance of auxiliary data type compared to algorithmic progress. Our experimental results also show that standard zero-shot learning approaches generalize poorly across categories of classes.
TokenCLIP: Token-wise Prompt Learning for Zero-shot Anomaly Detection
Zhejiang University（浙江大学）；Tencent YouTu Lab（腾讯YouTu实验室）；Singapore Management University（新加坡管理学院）；Qilu University of Technology (Shandong Academy of Sciences)（青岛科技大学（山东科学院））
Institute of Informatics Research, University of Alicante（阿利坎特大学信息学研究所）
One of the main issues related to unsupervised machine learning is the cost of processing and extracting useful information from large datasets. In this work, we propose a classifier ensemble based on the transferable learning capabilities of the CLIP neural network architecture in multimodal environments (image and text) from social media. For this purpose, we used the InstaNY100K dataset and proposed a validation approach based on sampling techniques. Our experiments, based on image classification tasks according to the labels of the Places dataset, are performed by first considering only th…
Graduate School of Information Science and Technology, Hokkaido University（北海道大学信息科学与技术研究院）；Faculty of Information Science and Technology, Hokkaido University（北海道大学信息科学与技术学院）
FILM: How can Few-Shot Image Classification Benefit from Pre-Trained Language Models?
Qing Yuan Research Institute（清源研究院）；Shanghai Jiao Tong University（上海交通大学）；College of Intelligence and Computing, Tianjin University（天津大学智能与计算学部）；Microsoft Research Asia（微软亚洲研究院）
Classifying a visual concept merely from its associated online textual source, such as a Wikipedia article, is an attractive research topic in zero-shot learning because it alleviates the burden of manually collecting semantic attributes. Several recent works have pursued this approach by exploring various ways of connecting the visual and text domains. This paper revisits this idea by stepping further to consider one important factor: the textual representation is usually too noisy for the zero-shot learning application. This consideration motivates us to design a simple-but-effective zero-sh…
Hacettepe University（哈斯特帕大学）；Middle East Technical University（中东技术大学）
We introduce the problem of zero-shot sign language recognition (ZSSLR), where the goal is to leverage models learned over the seen sign class examples to recognize the instances of unseen signs. To this end, we propose to utilize the readily available descriptions in sign language dictionaries as an intermediate-level semantic representation for knowledge transfer. We introduce a new benchmark dataset called ASL-Text that consists of 250 sign language classes and their accompanying textual descriptions. Compared to the ZSL datasets in other domains (such as object recognition), our dataset co…
This paper tackles the problem of zero-shot sign language recognition (ZSSLR), where the goal is to leverage models learned over the seen sign classes to recognize the instances of unseen sign classes. In this context, readily available textual sign descriptions and attributes collected from sign language dictionaries are utilized as semantic class representations for knowledge transfer. For this novel problem setup, we introduce three benchmark datasets with their accompanying textual and attribute descriptions to analyze the problem in detail. Our proposed approach builds spatiotemporal mode…
Pontificia Universidad Católica de Chile（智利天主教大学）；Samsung AI Centre Cambridge（三星人工智能中心剑桥）；Stanford University（斯坦福大学）；KU Leuven（鲁汶大学）；Facebook AI（Facebook 人工智能）
Recently, few-shot video classification has received an increasing interest. Current approaches mostly focus on effectively exploiting the temporal dimension in videos to improve learning under low data regimes. However, most works have largely ignored that videos are often accompanied by rich textual descriptions that can also be an essential source of information to handle few-shot recognition cases. In this paper, we propose to leverage these human-provided textual descriptions as privileged information when training a few-shot video classification model. Specifically, we formulate a text-b…
Zero-shot learning methods rely on fixed visual and semantic embeddings, extracted from independent vision and language models, both pre-trained for other large-scale tasks. This is a weakness of current zero-shot learning frameworks as such disjoint embeddings fail to adequately associate visual and textual information to their shared semantic content. Therefore, we propose to learn semantically grounded and enriched visual information by computing a joint image and text model with a two-stream network on a proxy task. To improve this alignment between image and textual representations, provi…
State Key Laboratory of Electromechanical Integrated Manufacturing of High-Performance Electronic Equipments（高性能电子设备机电一体化制造国家重点实验室）；Center for Complex Systems（复杂系统中心）；School of Mechano-Electronic Engineering（机械电子工程学院）；Xidian University（西安电子科技大学）
Department of CSE, IIT Kharagpur, India（印度理工学院卡格普尔分校计算机科学与工程系）；Department of ECE, Duke University, USA（美国杜克大学电子与计算机工程系）；Department of CSA, IISER Kolkata, India（印度科学教育与研究学院加尔各答分校计算机科学与应用系）
Most existing algorithms for cross-modal Information Retrieval are based on a supervised train-test setup, where a model learns to align the mode of the query (e.g., text) to the mode of the documents (e.g., images) from a given training set. Such a setup assumes that the training set contains an exhaustive representation of all possible classes of queries. In reality, a retrieval model may need to be deployed on previously unseen classes, which implies a zero-shot IR setup. In this paper, we propose a novel GAN-based model for zero-shot text to image retrieval. When given a textual descriptio…
On Aligning Hierarchical Standardized Embedding for Audio-visual Generalized Zero-shot Learning
Southeast University（东南大学）；The University of Hong Kong（香港大学）；Beijing Institute of Technology（北京理工大学）；Key Laboratory of New Generation Artificial Intelligence Technology and Its Interdisciplinary Applications (Southeast University), Ministry of Education（新一代人工智能技术及其跨学科应用重点实验室（东南大学），教育部）；School of Computer Science and Engineering, Southeast University（东南大学计算机科学与工程学院）
One of the main challenges in Zero-Shot Learning of visual categories is gathering semantic attributes to accompany images. Recent work has shown that learning from textual descriptions, such as Wikipedia articles, avoids the problem of having to explicitly define these attributes. We present a new model that can classify unseen categories from their textual description. Specifically, we use text features to predict the output weights of both the convolutional and the fully connected layers in a deep convolutional neural network (CNN). We take advantage of the architecture of CNNs and learn fe…
FG-MDM: Towards Zero-Shot Human Motion Generation via ChatGPT-Refined Descriptions
Nanjing University of Science and Technology（南京理工大学）；Shandong University（山东大学）；Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；Beijing Normal University（北京师范大学）
National University of Ireland, Galway（爱尔兰国立大学戈尔韦分校）
Machine Learning (ML) techniques for image classification routinely require many labelled images for training the model and while testing, we ought to use images belonging to the same domain as those used for training. In this paper, we overcome the two main hurdles of ML, i.e. scarcity of data and constrained prediction of the classification model. We do this by introducing a visual classifier which uses a concept of transfer learning, namely Zero-Shot Learning (ZSL), and standard Natural Language Processing techniques. We train a classifier by mapping labelled images to their textual descrip…
School of Computing Sciences, University of East Anglia（东英吉利大学计算科学学院）；Malong Technologies Co., Ltd.（码隆科技有限公司）；Future Media Center, University of Electronic Science and Technology of China（电子科技大学未来媒体中心）
Cross-modal hashing is usually regarded as an effective technique for large-scale textual-visual cross retrieval, where data from different modalities are mapped into a shared Hamming space for matching. Most of the traditional textual-visual binary encoding methods only consider holistic image representations and fail to model descriptive sentences. This renders existing methods inappropriate to handle the rich semantics of informative cross-modal data for quality textual-visual search tasks. To address the problem of hashing cross-modal data with semantic-rich cues, in this paper, a novel in…
Prototype learning and decoder construction are the keys for few-shot segmentation. However, existing methods use only a single prototype generation mode, which can not cope with the intractable problem of objects with various scales. Moreover, the one-way forward propagation adopted by previous methods may cause information dilution from registered features during the decoding process. In this research, we propose a rich prototype generation module (RPGM) and a recurrent prediction enhancement module (RPEM) to reinforce the prototype learning paradigm and build a unified memory-augmented deco…
FB-CLIP: Fine-Grained Zero-Shot Anomaly Detection with Foreground-Background Disentanglement
Xi’an Institute of Optics and Precision Mechanics, Chinese Academy of Sciences（西安光学精密机械研究所，中国科学院）；University of Chinese Academy of Sciences（中国科学院大学）；Xi’an Jiaotong University（西安交通大学）；Zhongnan Hospital of Wuhan University（武汉大学中南医院）
Expanding Zero-Shot Object Counting with Rich Prompts
Hubei Key Laboratory of Transportation Internet of Things, Wuhan University of Technology（湖北交通物联网重点实验室，武汉理工大学）；School of Computing and Information Systems, Singapore Management University（新加坡管理大学计算机与信息系统学院）；School of Computer Science, Wuhan University（武汉大学计算机学院）；School of Navigation, Wuhan University of Technology（武汉理工大学导航学院）；School of Computer Science, Peking University（北京大学计算机学院）
