Global Semantic Description of Objects based on Prototype Theory
Universidade Federal de Minas Gerais（米纳斯吉拉斯联邦大学）
Omar Vidal Pino, Erickson Rangel Nascimento, Mario Fernando Montenegro Campos
备注：
Content: 24 pages (22 + 2 reference) with 15 Figures and 3 Tables. In the future, a new version will be updated with other experiments and results (and a journal reference if applicable)
In this paper, we introduce a novel semantic description approach inspired on Prototype Theory foundations. We propose a Computational Prototype Model (CPM) that encodes and stores the central semantic meaning of objects category: the semantic prototype. Also, we introduce a Prototype-based Description Model that encodes the semantic meaning of an object while describing its features using our CPM model. Our description method uses semantic prototypes computed by CNN-classifications models to create discriminative signatures that describe an object highlighting its most distinctive features within the category. Our experiments show that: i) our CPM model (semantic prototype + distance metric) is able to describe the internal semantic structure of objects categories; ii) our semantic distance metric can be understood as the object visual typicality score within a category; iii) our descriptor encoding is semantically interpretable and significantly outperforms other image global encodings in clustering and classification tasks.
Tsinghua University（清华大学）；Tencent Youtu Lab（腾讯优图实验室）；University College London（伦敦大学学院）
Few-shot semantic segmentation aims to segment novel-class objects in a query image with only a few annotated examples in support images. Most of advanced solutions exploit a metric learning framework that performs segmentation through matching each pixel to a learned foreground prototype. However, this framework suffers from biased classification due to incomplete construction of sample pairs with the foreground prototype only. To address this issue, in this paper, we introduce a complementary self-contrastive task into few-shot semantic segmentation. Our new model is able to associate the pi…
Shanghai Jiao Tong University（上海交通大学）；University of Maryland（马里兰大学）；Virginia Tech（弗吉尼亚理工大学）；Shanghai AI Laboratory（上海人工智能实验室）
3D point cloud semantic segmentation is one of the fundamental tasks for 3D scene understanding and has been widely used in the metaverse applications. Many recent 3D semantic segmentation methods learn a single prototype (classifier weights) for each semantic class, and classify 3D points according to their nearest prototype. However, learning only one prototype for each class limits the model's ability to describe the high variance patterns within a class. Instead of learning a single prototype for each class, in this paper, we propose to use an adaptive number of prototypes to dynamically d…
Semantic segmentation methods can not directly identify abnormal objects in images. Anomaly Segmentation algorithm from this realistic setting can distinguish between in-distribution objects and Out-Of-Distribution (OOD) objects and output the anomaly probability for pixels. In this paper, a Prototype Guided Anomaly segmentation Network (PGAN) is proposed to extract semantic prototypes for in-distribution training data from limited annotated images. In the model, prototypes are used to model the hierarchical category semantic information and distinguish OOD pixels. The proposed PGAN model incl…
University of Chinese Academy of Sciences（中国科学院大学）
Few-shot segmentation is challenging because objects within the support and query images could significantly differ in appearance and pose. Using a single prototype acquired directly from the support image to segment the query image causes semantic ambiguity. In this paper, we propose prototype mixture models (PMMs), which correlate diverse image regions with multiple prototypes to enforce the prototype-based semantic representation. Estimated by an Expectation-Maximization algorithm, PMMs incorporate rich channel-wised and spatial semantics from limited support images. Utilized as representat…
Shanghai Institute of Microsystem and Information Technology, Chinese Academy of Sciences（中国科学院上海微系统与信息技术研究所）；University of Chinese Academy of Sciences（中国科学院大学）；ShanghaiTech University（上海科技大学）
Few-shot semantic segmentation aims to learn to segment new object classes with only a few annotated examples, which has a wide range of real-world applications. Most existing methods either focus on the restrictive setting of one-way few-shot segmentation or suffer from incomplete coverage of object regions. In this paper, we propose a novel few-shot semantic segmentation framework based on the prototype representation. Our key idea is to decompose the holistic class representation into a set of part-aware prototypes, capable of capturing diverse and fine-grained object features. In addition,…
Few-Shot Medical Image Segmentation with High-Fidelity Prototypes
University of Shanghai for Science and Technology（上海理工大学）；Universität Hamburg（汉堡大学）；Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences（中国科学院深圳先进技术研究院）；University of Electronic Science and Technology of China（电子科技大学）；University of Surrey（萨里大学）
Although there is significant progress in supervised semantic segmentation, it remains challenging to deploy the segmentation models to unseen domains due to domain biases. Domain adaptation can help in this regard by transferring knowledge from a labeled source domain to an unlabeled target domain. Previous methods typically attempt to perform the adaptation on global features, however, the local semantic affiliations accounting for each pixel in the feature space are often ignored, resulting in less discriminability. To solve this issue, we propose a novel semantic prototype-based contrastiv…
Carnegie Mellon University（卡内基梅隆大学）；Mohamed bin Zayed University of Artificial Intelligence（穆罕默德·本·扎耶德人工智能大学）；Huazhong University of Science and Technology（华中科技大学）；University of Sydney（悉尼大学）；Tencent AI LAB（腾讯人工智能实验室）；AI 3 Institute, Fudan University（复旦大学人工智能创新与产业研究院）
Prototype-Guided Curriculum Learning for Zero-Shot Learning
School of Electronic Information and Communications, Huazhong University of Science and Technology（电子信息与通信学院，华中科技大学）；Mohamed bin Zayed University of AI（马尔代夫人工智能大学）；School of Computer Science and Engineering, Nanjing University of Science and Technology（计算机科学与工程学院，南京理工大学）；School of Computer Science, The University of Sydney（计算机科学学院，悉尼大学）
Semantically Guided Dynamic Visual Prototype Refinement for Compositional Zero-Shot Learning
National Key Laboratory of Radar Signal Processing, Xidian University（雷达信号处理国家级重点实验室，西安电子科技大学）；Research Institute of Systems Engineering, Academy of Military Science（系统工程研究所，军事科学院）
Project and Mix: Task-Semantic Prototypes for Few-Shot Image Classification
Department of Computer Science, Universitat Autònoma de Barcelona, Spain（巴塞罗那自治大学计算机科学系）；Computer Vision Center, Barcelona, Spain（巴塞罗那计算机视觉中心）；Media Integration and Communication Center, University of Florence, Italy（佛罗伦萨大学媒体整合与传播中心）；Bernoulli Institute, University of Groningen, the Netherlands（格罗宁根大学伯努利学院）；IDEAS Research Institute, Poland（波兰IDEAS研究所）；ESAT-PSI, KU Leuven, Belgium（比利时KU莱顿大学ESAT-PSI）
Beihang University（北京航空航天大学）；Beijing University of Posts and Telecommunications（北京邮电大学）
Zero-shot learning (ZSL) aims to recognize unseen classes by exploiting semantic descriptions shared between seen classes and unseen classes. Current methods show that it is effective to learn visual-semantic alignment by projecting semantic embeddings into the visual space as class prototypes. However, such a projection function is only concerned with seen classes. When applied to unseen classes, the prototypes often perform suboptimally due to domain shift. In this paper, we propose to learn prototypes via placeholders, termed LPL, to eliminate the domain shift between seen and unseen classe…
Multigrain-aware Semantic Prototype Scanning and Tri-Token Prompt Learning Embraced High-Order RWKV for Pan-Sharpening
Shenzhen Campus of Sun Yat-sen University（中山大学深圳校区）；Yunnan Normal University, Ministry of Education（云南师范大学教育部）；Xidian University（西安电子科技大学）；University of Science and Technology of China（中国科学技术大学）；The Hong Kong University of Science and Technology（香港理工大学）；MoE Key Laboratory of Information Technology（教育部信息科学技术重点实验室）
FedSaaS: Class-Consistency Federated Semantic Segmentation via Global Prototype Supervision and Local Adversarial Harmonization
Shandong Computer Science Center, Qilu University of Technology (Shandong Academy of Sciences)（山东省计算中心（齐鲁工业大学（山东省科学院）））；Shandong Fundamental Research Center for Computer Science（山东省计算机科学基础研究中心）；College of Control Science and Engineering, Zhejiang University（浙江大学控制科学与工程学院）
Semantic-embedded Similarity Prototype for Scene Recognition
Shandong University（山东大学）；School of Control Science and Engineering（控制科学与工程学院）；Engineering Research Center of Intelligent Unmanned System, Ministry of Education（教育部智能无人系统工程研究中心）
Politecnico di Torino（都灵理工大学）；Italian Institute of Technology（意大利理工学院）；University of Tübingen（图宾根大学）；ETH Zurich（苏黎世联邦理工学院）
Semantic segmentation models have two fundamental weaknesses: i) they require large training sets with costly pixel-level annotations, and ii) they have a static output space, constrained to the classes of the training set. Toward addressing both problems, we introduce a new task, Incremental Few-Shot Segmentation (iFSS). The goal of iFSS is to extend a pretrained segmentation model with new classes from few annotated images and without access to old training data. To overcome the limitations of existing models iniFSS, we propose Prototype-based Incremental Few-Shot Segmentation (PIFS) that co…
The University of Adelaide（阿德莱德大学）；Australian Institute for Machine Learning（澳大利亚机器学习研究所）；Northeastern University（东北大学）；Huawei Noah’s Ark Lab（华为诺亚方舟实验室）
Semi-supervised semantic segmentation requires the model to effectively propagate the label information from limited annotated images to unlabeled ones. A challenge for such a per-pixel prediction task is the large intra-class variation, i.e., regions belonging to the same class may exhibit a very different appearance even in the same picture. This diversity will make the label propagation hard from pixels to pixels. To address this problem, we propose a novel approach to regularize the distribution of within-class features to ease label propagation difficulty. Specifically, our approach encou…
ETH Zurich（苏黎世联邦理工学院）；University of Technology Sydney（悉尼科技大学）
Prevalent semantic segmentation solutions, despite their different network designs (FCN based or attention based) and mask decoding strategies (parametric softmax based or pixel-query based), can be placed in one category, by considering the softmax weights or query vectors as learnable class prototypes. In light of this prototype view, this study uncovers several limitations of such parametric segmentation regime, and proposes a nonparametric alternative based on non-learnable prototypes. Instead of prior methods learning a single weight/query vector for each class in a fully parametric manne…
Despite the great progress made by deep CNNs in image semantic segmentation, they typically require a large number of densely-annotated images for training and are difficult to generalize to unseen object categories. Few-shot segmentation has thus been developed to learn to perform segmentation from only a few annotated examples. In this paper, we tackle the challenging few-shot segmentation problem from a metric learning perspective and present PANet, a novel prototype alignment network to better utilize the information of the support set. Our PANet learns class-specific prototype representat…
We address the problem of few-shot semantic segmentation (FSS), which aims to segment novel class objects in a target image with a few annotated samples. Though recent advances have been made by incorporating prototype-based metric learning, existing methods still show limited performance under extreme intra-class object variations and semantically similar inter-class objects due to their poor feature representation. To tackle this problem, we propose a dual prototypical contrastive learning approach tailored to the FSS task to capture the representative semanticfeatures effectively. The main …
Tsinghua University（清华大学）；Tencent YouTu Lab（腾讯优图实验室）；University College London（伦敦大学学院）
Few-shot semantic segmentation aims to segment novel-class objects in a given query image with only a few labeled support images. Most advanced solutions exploit a metric learning framework that performs segmentation through matching each query feature to a learned class-specific prototype. However, this framework suffers from biased classification due to incomplete feature comparisons. To address this issue, we present an adaptive prototype representation by introducing class-specific and class-agnostic prototypes and thus construct complete sample pairs for learning semantic alignment with q…
School of Computer Science and Engineering, Nanjing University of Science and Technology（南京理工大学计算机科学与工程学院）；Inception Institute of Artificial Intelligence（Inception人工智能研究院）；School of Computer Science and Engineering, University of Electronic Science and Technology of China（电子科技大学计算机科学与工程学院）
One-shot semantic image segmentation aims to segment the object regions for the novel class with only one annotated image. Recent works adopt the episodic training strategy to mimic the expected situation at testing time. However, these existing approaches simulate the test conditions too strictly during the training process, and thus cannot make full use of the given label information. Besides, these approaches mainly focus on the foreground-background target class segmentation setting. They only utilize binary mask labels for training. In this paper, we propose to leverage the multi-class la…
ShanghaiTech University（上海科技大学）；University of Chinese Academy of Sciences（中国科学院大学）
Zero-shot learning (ZSL) aims to recognize objects of novel classes without any training samples of specific classes, which is achieved by exploiting the semantic information and auxiliary datasets. Recently most ZSL approaches focus on learning visual-semantic embeddings to transfer knowledge from the auxiliary datasets to the novel classes. However, few works study whether the semantic information is discriminative or not for the recognition task. To tackle such problem, we propose a coupled dictionary learning approach to align the visual-semantic structures using the class prototypes, wher…
Universidade Federal de Minas Gerais (UFMG)（米纳斯吉拉斯联邦大学）
In this paper, we introduce a novel approach for semantic description of object features based on the prototypicality effects of the Prototype Theory. Our prototype-based description model encodes and stores the semantic meaning of an object, while describing its features using the semantic prototype computed by CNN-classifications models. Our method uses semantic prototypes to create discriminative descriptor signatures that describe an object highlighting its most distinctive features within the category. Our experiments show that: i) our descriptor preserves the semantic information used by…
Visual-Augmented Dynamic Semantic Prototype for Generative Zero-Shot Learning
Huazhong University of Science and Technology (HUST)（华中科技大学）；Alibaba Group（阿里巴巴集团）；The University of Sydney（悉尼大学）；Mohamed bin Zayed University of AI（穆罕默德·本·扎耶德人工智能大学）；Australian National University（澳大利亚国立大学）；Linköping University（林雪平大学）
Few to Big: Prototype Expansion Network via Diffusion Learner for Point Cloud Few-shot Semantic Segmentation
School of Mathematics and Computational Science, Xiangtan University（湘潭大学数学与计算科学学院）；School of Automation and Electronics Information, Xiangtan University（湘潭大学自动化与电子信息学院）；School of Automation, Shanghai Jiao Tong University（上海交通大学自动化学院）
Decouple, Purify and Unite: Semantic-Structural Prototype Learning for Federated Medical Segmentation
Peking Union Medical College Hospital, Chinese Academy of Medical Sciences and Peking Union Medical College（北京协和医院，中国医学科学院北京协和医学院）；Beijing University of Posts and Telecommunications（北京邮电大学）；Nanyang Technological University（南洋理工大学）；Ant Group（蚂蚁集团）；Shanghai Innovation Institute（上海创新研究院）；Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；DAMO Academy, Alibaba Group（阿里巴巴达摩院）；Fourth Military Medical University（中国人民解放军空军军医大学）；Macao Polytechnic University（澳门理工大学）；Wuhan University（武汉大学）；The Hong Kong Polytechnic University（香港理工大学）
Prototype-Based Pseudo-Label Denoising for Source-Free Domain Adaptation in Remote Sensing Semantic Segmentation
College of Computer Science, Sichuan University（四川大学计算机科学学院）；College of Computer Science and Cyber Security, Chengdu University of Technology（成都理工大学计算机科学与网络安全学院）
Boundary-Refined Prototype Generation: A General End-to-End Paradigm for Semi-Supervised Semantic Segmentation
School of Artificial Intelligence, Beijing University of Posts and Telecommunications（北京邮电大学人工智能学院）；Beijing Key Laboratory of Network System and Network Culture（网络系统与网络文化北京市重点实验室）
University of Chinese Academy of Sciences（中国科学院大学）；Wuhan University（武汉大学）
Few-shot segmentation of point cloud remains a challenging task, as there is no effective way to convert local point cloud information to global representation, which hinders the generalization ability of point features. In this study, we propose a bidirectional feature globalization (BFG) approach, which leverages the similarity measurement between point features and prototype vectors to embed global perception to local point features in a bidirectional fashion. With point-to-prototype globalization (Po2PrG), BFG aggregates local point features to prototypes according to similarity weights fr…
Institute of Computing Technology, CAS（中国科学院计算技术研究所）；University of Chinese Academy of Sciences（中国科学院大学）；Beijing Academy of Artificial Intelligence（北京智源人工智能研究院）
Teaching machines to recognize a new category based on few training samples especially only one remains challenging owing to the incomprehensive understanding of the novel category caused by the lack of data. However, human can learn new classes quickly even given few samples since human can tell what discriminative features should be focused on about each category based on both the visual and semantic prior knowledge. To better utilize those prior knowledge, we propose the SEmantic Guided Attention (SEGA) mechanism where the semantic knowledge is used to guide the visual perception in a top-d…
The deep CNNs in image semantic segmentation typically require a large number of densely-annotated images for training and have difficulties in generalizing to unseen object categories. Therefore, few-shot segmentation has been developed to perform segmentation with just a few annotated examples. In this work, we tackle the few-shot segmentation using a self-regularized prototypical network (SRPNet) based on prototype extraction for better utilization of the support information. The proposed SRPNet extracts class-specific prototype representations from support images and generates segmentation…
Beijing Jiaotong University（北京交通大学）；University of Rochester（罗切斯特大学）
Zero-Shot Learning (ZSL) has received extensive attention and successes in recent years especially in areas of fine-grained object recognition, retrieval, and image captioning. Key to ZSL is to transfer knowledge from the seen to the unseen classes via auxiliary semantic prototypes (e.g., word or attribute vectors). However, the popularly learned projection functions in previous works cannot generalize well due to non-visual components included in semantic prototypes. Besides, the incompleteness of provided prototypes and captured images has less been considered by the state-of-the-art approac…
Northwestern Polytechnical University（西北工业大学）；Mohamed bin Zayed University of Artificial Intelligence（穆罕默德·本·扎耶德人工智能大学）
Few-shot semantic segmentation aims to segment the target objects in query under the condition of a few annotated support images. Most previous works strive to mine more effective category information from the support to match with the corresponding objects in query. However, they all ignored the category information gap between query and support images. If the objects in them show large intra-class diversity, forcibly migrating the category information from the support to the query is ineffective. To solve this problem, we are the first to introduce an intermediate prototype for mining both d…
Politecnico di Torino（都灵理工大学）；Italian Institute of Technology（意大利理工学院）；University of Tübingen（图宾根大学）
Traditional semantic segmentation methods can recognize at test time only the classes that are present in the training set. This is a significant limitation, especially for semantic segmentation algorithms mounted on intelligent autonomous systems, deployed in realistic settings. Regardless of how many classes the system has seen at training time, it is inevitable that unexpected, unknown objects will appear at test time. The failure in identifying such anomalies may lead to incorrect, even dangerous behaviors of the autonomous agent equipped with such segmentation model when deployed in the r…
Zhejiang University（浙江大学）；Tianjin University（天津大学）；University of Warwick（华威大学）；Binghamton University（宾汉姆顿大学）
We introduce a simple yet effective episode-based training framework for zero-shot learning (ZSL), where the learning system requires to recognize unseen classes given only the corresponding class semantics. During training, the model is trained within a collection of episodes, each of which is designed to simulate a zero-shot classification task. Through training multiple episodes, the model progressively accumulates ensemble experiences on predicting the mimetic unseen classes, which will generalize well on the real unseen classes. Based on this training framework, we propose a novel generat…
Supervised Classification Heads as Semantic Prototypes: Unlocking Vision-Language Alignment via Weight Recycling
Department of Computer Science and Artificial Intelligence, DaSCI Institute, University of Granada, Granada, Spain（计算机科学与人工智能系，DaSCI研究所，格拉纳达大学，格拉纳达，西班牙）；Department of Mathematics ``Tullio Levi-Civita'', University of Padova, Padova, Italy（托里利-西维塔数学系，帕多瓦大学，帕多瓦，意大利）
