ZeroDiff++: Substantial Unseen Visual-semantic Correlation in Zero-shot Learning
UCAS-Terminus AI Lab, University of Chinese Academy of Sciences（中国科学院大学Terminus AI实验室，中国科学院大学）；School of Computer Science, the University of Nottingham（诺丁汉大学计算机学院）；Southeast University（东南大学）；hi-lab of Xiaohongshu Inc（小红书公司hi实验室）
Zihan Ye, Shreyank N Gowda, Kaile Du, Weijian Luo, Ling Shao
Zero-shot Learning (ZSL) enables classifiers to recognize classes unseen during training, commonly via generative two stage methods: (1) learn visual semantic correlations from seen classes; (2) synthesize unseen class features from semantics to train classifiers. In this paper, we identify spurious visual semantic correlations in existing generative ZSL worsened by scarce seen class samples and introduce two metrics to quantify spuriousness for seen and unseen classes. Furthermore, we point out a more critical bottleneck: existing unadaptive fully noised generators produce features disconnected from real test samples, which also leads to the spurious correlation. To enhance the visual-semantic correlations on both seen and unseen classes, we propose ZeroDiff++, a diffusion-based generative framework. In training, ZeroDiff++ uses (i) diffusion augmentation to produce diverse noised samples, (ii) supervised contrastive (SC) representations for instance level semantics, and (iii) multi view discriminators with Wasserstein mutual learning to assess generated features. At generation time, we introduce (iv) Diffusion-based Test time Adaptation (DiffTTA) to adapt the generator using pseudo label reconstruction, and (v) Diffusion-based Test time Generation (DiffGen) to trace the diffusion denoising path and produce partially synthesized features that connect real and generated data, and mitigates data scarcity further. Extensive experiments on three ZSL benchmarks demonstrate that ZeroDiff++ not only achieves significant improvements over existing ZSL methods but also maintains robust performance even with scarce training data. Code would be available.
ZeroDiff: Solidified Visual-Semantic Correlation in Zero-Shot Learning
Xi’an Jiaotong-Liverpool University（西交利物浦大学）；University of Liverpool（利物浦大学）；University of Nottingham（诺丁汉大学）；Mohamed bin Zayed University of Artificial Intelligence（穆罕默德·本·扎耶德人工智能大学）；Westlake University（西湖大学）；Duke Kunshan University（昆山杜克大学）
Accurate and Efficient Object Pose Estimation via the Aggregation of Diffusion Features
Shenyang Institute of Automation, Chinese Academy of Sciences（中国科学院沈阳自动化研究所）；Institutes for Robotics and Intelligent Manufacturing, Chinese Academy of Sciences（中国科学院机器人与智能制造创新研究院）；University of Chinese Academy of Sciences（中国科学院大学）；Oosto（奥斯特公司）
Improved Feature Generating Framework for Transductive Zero-shot Learning
University of Liverpool（利物浦大学）；Westlake University（西湖大学）；Mohamed bin Zayed University of Artificial Intelligence（穆罕默德·本·扎耶德人工智能大学）；Duke Kunshan University（昆山杜克大学）
Anomaly Heterogeneity Learning for Open-set Supervised Anomaly Detection
School of Computing and Information Systems, Singapore Management University（新加坡管理大学计算与信息系统学院）；Australian Institute for Machine Learning, University of Adelaide（阿德莱德大学澳大利亚机器学习研究所）；Harvard University（哈佛大学）
Learning Adversarial Semantic Embeddings for Zero-Shot Recognition in Open Worlds
Beihang University（北京航空航天大学）；Singapore Management University（新加坡管理大学）；Wuhan University of Technology（武汉理工大学）；Institute of Semiconductors, Chinese Academy of Sciences（中国科学院半导体研究所）
City University of Hong Kong（香港城市大学）；Microsoft Cloud+AI（微软云与人工智能部门）；PCG, Tencent（腾讯平台与内容事业群）；Shenzhen University（深圳大学）；NLPR, CASIA（中国科学院自动化研究所模式识别国家重点实验室）；Deepwise AI Lab（深睿人工智能实验室）
To recognize objects of the unseen classes, most existing Zero-Shot Learning(ZSL) methods first learn a compatible projection function between the common semantic space and the visual space based on the data of source seen classes, then directly apply it to the target unseen classes. However, in real scenarios, the data distribution between the source and target domain might not match well, thus causing the well-known \textbf{domain shift} problem. Based on the observation that visual features of test instances can be separated into different clusters, we propose a new visual structure constra…
DistillFSS: Synthesizing Few-Shot Knowledge into a Lightweight Segmentation Model
Department of Computer Science, University of Bari Aldo Moro（巴里阿尔多·莫罗大学计算机科学系）；Jheronimus Academy of Data Science (JADS)（赫尔蒙尼乌斯数据科学学院）；Eindhoven University of Technology (TU/e)（埃因霍温理工大学）
Point to Span: Zero-Shot Moment Retrieval for Navigating Unseen Hour-Long Videos
Chung-Ang University（中央大学）；Korea Advanced Institute of Science and Technology (KAIST)（韩国科学技术院（KAIST））；Electronics and Telecommunications Research Institute (ETRI)（电子通信研究院（ETRI））
School of Cyber Science and Engineering, Southeast University（东南大学信息科学与工程学院）；Engineering Research Center of Blockchain Application, Supervision and Management (Southeast University), Ministry of Education（教育部区块链应用、监督与管理工程研究中心（东南大学））；Purple Mountain Laboratories, Nanjing（紫金山实验室（南京））；School of Computer Science and Engineering, Southeast University（东南大学计算机科学与工程学院）
Object recognition in unseen indoor environments remains a challenging problem for visual perception of mobile robots. In this letter, we propose the use of topologically persistent features, which rely on the objects' shape information, to address this challenge. In particular, we extract two kinds of features, namely, sparse persistence image (PI) and amplitude, by applying persistent homology to multi-directional height function-based filtrations of the cubical complexes representing the object segmentation maps. The features are then used to train a fully connected network for recognition.…
Peking University（北京大学）；SenseTime Research（商汤科技研究院）；CRIL - University of Artois & CNRS（法国国立计算机与自动化研究所 - 阿图瓦大学与法国国家科学研究中心）；University of Essex（埃塞克斯大学）；Cardiff University（卡迪夫大学）
Multi-label few-shot image classification (ML-FSIC) is the task of assigning descriptive labels to previously unseen images, based on a small number of training examples. A key feature of the multi-label setting is that images often have multiple labels, which typically refer to different regions of the image. When estimating prototypes, in a metric-based setting, it is thus important to determine which regions are relevant for which labels, but the limited amount of training data makes this highly challenging. As a solution, in this paper we propose to use word embeddings as a form of prior k…
Zero-shot learning (ZSL) has been shown to be a promising approach to generalizing a model to categories unseen during training by leveraging class attributes, but challenges still remain. Recently, methods using generative models to combat bias towards classes seen during training have pushed the state of the art of ZSL, but these generative models can be slow or computationally expensive to train. Additionally, while many previous ZSL methods assume a one-time adaptation to unseen classes, in reality, the world is always changing, necessitating a constant adjustment for deployed models. Mode…
We address the problem of learning fine-grained cross-modal representations. We propose an instance-based deep metric learning approach in joint visual and textual space. The key novelty of this paper is that it shows that using per-image semantic supervision leads to substantial improvement in zero-shot performance over using class-only supervision. On top of that, we provide a probabilistic justification for a metric rescaling approach that solves a very common problem in the generalized zero-shot learning setting, i.e., classifying test images from unseen classes as one of the classes seen …
