Evaluation Measures for Hierarchical Classification: a unified view and novel approaches
National Center for Scientific Research “Demokritos”（国家科学研究中心“德谟克利特”）；Athens University of Economics and Business（雅典经济与商业大学）；Laboratoire d’Informatique de Grenoble（格勒诺布尔信息学实验室）；Univesité Joseph Fourier（约瑟夫·傅立叶大学）
Aris Kosmopoulos, Ioannis Partalas, Eric Gaussier, Georgios Paliouras, Ion Androutsopoulos
Hierarchical classification addresses the problem of classifying items into a hierarchy of classes. An important issue in hierarchical classification is the evaluation of different classification algorithms, which is complicated by the hierarchical relations among the classes. Several evaluation measures have been proposed for hierarchical classification using the hierarchy in different ways. This paper studies the problem of evaluation in hierarchical classification by analyzing and abstracting the key components of the existing performance measures. It also proposes two alternative generic views of hierarchical evaluation and introduces two corresponding novel measures. The proposed measures, along with the state-of-the art ones, are empirically tested on three large datasets from the domain of text classification. The empirical results illustrate the undesirable behavior of existing approaches and how the proposed methods overcome most of these methods across a range of cases.
University of Electronic Science and Technology of China（电子科技大学）；Tencent Inc.（腾讯公司）；Harbin Institute of Technology, Shenzhen（哈尔滨工业大学（深圳））；Peng Cheng National Lab（鹏城国家实验室）
Hierarchical clustering recursively partitions data at an increasingly finer granularity. In real-world applications, multi-view data have become increasingly important. This raises a less investigated problem, i.e., multi-view hierarchical clustering, to better understand the hierarchical structure of multi-view data. To this end, we propose a novel neural network-based model, namely Contrastive Multi-view Hyperbolic Hierarchical Clustering (CMHHC). It consists of three components, i.e., multi-view alignment learning, aligned feature similarity learning, and continuous hyperbolic hierarchical…
Indian Institute of Technology Kanpur（坎普尔印度理工学院）；Microsoft Research（微软研究院）
Hierarchical classification is supervised multi-class classification problem over the set of class labels organized according to a hierarchy. In this report, we study the work by Ramaswamy et. al. on hierarchical classification over symmetric tree distance loss. We extend the consistency of hierarchical classification algorithm over asymmetric tree distance loss. We design a $\mathcal{O}(nk\log{}n)$ algorithm to find Bayes optimal classification for a k-ary tree as a hierarchy. We show that under reasonable assumptions over asymmetric loss function, the Bayes optimal classification over this a…
We propose methods that enable efficient hierarchical classification in parallel. Our methods transform a batch of classification scores and labels, corresponding to given nodes in a semantic tree, to scores and labels corresponding to all nodes in the ancestral paths going down the tree to every given node, relying only on tensor operations that execute efficiently on hardware accelerators. We implement our methods and test them on current hardware accelerators with a tree incorporating all English-language synsets in WordNet 3.0, spanning 117,659 classes in 20 levels of depth. We transform b…
Hierarchical Mutual Information Analysis: Towards Multi-view Clustering in The Wild
OPPO Research Institute（OPPO研究院）；Xinchuang Haihe Laboratory（信创海河实验室）；Institute of Computing Technology, Chinese Academy of Sciences（中国科学院计算技术研究所）；InnoPeak Technology, Inc（InnoPeak Technology公司）；Stony Brook University（石溪大学）
University of Illinois at Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）
Hierarchical text classification, which aims to classify text documents into a given hierarchy, is an important task in many real-world applications. Recently, deep neural models are gaining increasing popularity for text classification due to their expressive power and minimum requirement for feature engineering. However, applying deep neural networks for hierarchical text classification remains challenging, because they heavily rely on a large amount of training data and meanwhile cannot easily determine appropriate levels of documents in the hierarchical setting. In this paper, we propose a…
Efficient distributed numerical word representation models (word embeddings) combined with modern machine learning algorithms have recently yielded considerable improvement on automatic document classification tasks. However, the effectiveness of such techniques has not been assessed for the hierarchical text classification (HTC) yet. This study investigates the application of those models and algorithms on this specific problem by means of experimentation and analysis. We trained classification models with prominent machine learning algorithm implementations---fastText, XGBoost, SVM, and Kera…
Hong Kong University of Science and Technology (Guangzhou)（香港科技大学（广州））；Hong Kong University of Science and Technology（香港科技大学）；Tsinghua University（清华大学）；Tencent AI Lab（腾讯人工智能实验室）
Node classification and graph classification are two graph learning problems that predict the class label of a node and the class label of a graph respectively. A node of a graph usually represents a real-world entity, e.g., a user in a social network, or a document in a document citation network. In this work, we consider a more challenging but practically useful setting, in which a node itself is a graph instance. This leads to a hierarchical graph perspective which arises in many domains such as social network, biological network and document collection. We study the node classification pro…
Several real problems ranging from text classification to computational biology are characterized by hierarchical multi-label classification tasks. Most of the methods presented in literature focused on tree-structured taxonomies, but only few on taxonomies structured according to a Directed Acyclic Graph (DAG). In this contribution novel classification ensemble algorithms for DAG-structured taxonomies are introduced. In particular Hierarchical Top-Down (HTD-DAG) and True Path Rule (TPR-DAG) for DAGs are presented and discussed.
Sejong University（世宗大学）；Electronics and Telecommunications Research Institute（电子通信研究院）
We propose an effective method to improve Protein Function Prediction (PFP) utilizing hierarchical features of Gene Ontology (GO) terms. Our method consists of a language model for encoding the protein sequence and a Graph Convolutional Network (GCN) for representing GO terms. To reflect the hierarchical structure of GO to GCN, we employ node(GO term)-wise representations containing the whole hierarchical information. Our algorithm shows effectiveness in a large-scale graph by expanding the GO graph compared to previous models. Experimental results show that our method outperformed state-of-th…
Hierarchical Classification (HC) is a supervised learning problem where unlabeled instances are classified into a taxonomy of classes. Several methods that utilize the hierarchical structure have been developed to improve the HC performance. However, in most cases apriori defined hierarchical structure by domain experts is inconsistent; as a consequence performance improvement is not noticeable in comparison to flat classification methods. We propose a scalable data-driven filter based rewiring approach to modify an expert-defined hierarchy. Experimental comparisons of top-down HC with our mod…
Ghent University（根特大学）；LMU Munich（慕尼黑大学）；Poznań University of Technology（波兹南理工大学）；Yahoo! Research（雅虎研究院）
Set-valued prediction is a well-known concept in multi-class classification. When a classifier is uncertain about the class label for a test instance, it can predict a set of classes instead of a single class. In this paper, we focus on hierarchical multi-class classification problems, where valid sets (typically) correspond to internal nodes of the hierarchy. We argue that this is a very strong restriction, and we propose a relaxation by introducing the notion of representation complexity for a predicted set. In combination with probabilistic classifiers, this leads to a challenging inference…
Hierarchical Generalized Category Discovery for Brain Tumor Classification in Digital Pathology
Institute of Clinical Epidemiology, Public Health, Health Economics, Medical Statistics and Informatics（临床流行病学、公共卫生、健康经济学、医学统计与信息学研究所）；Institute of Neuropathology and Neuromolecular Pathology（神经病理学与神经分子病理学研究所）
Leveraging Taxonomy and LLMs for Improved Multimodal Hierarchical Classification
University of New South Wales（新南威尔士大学）；Deakin University（迪肯大学）；University of Southampton（南安普顿大学）；Macquarie University（麦考瑞大学）；Technology Innovation Institute（技术创新研究院）；MBZUAI（穆罕默德·本·扎耶德人工智能大学）
Department of Computer Science, University of Virginia（弗吉尼亚大学计算机科学系）；Department of System and Information Engineering, University of Virginia（弗吉尼亚大学系统与信息工程系）；Data Science Institute, University of Virginia（弗吉尼亚大学数据科学研究所）
The continually increasing number of documents produced each year necessitates ever improving information processing methods for searching, retrieving, and organizing text. Central to these information processing methods is document classification, which has become an important application for supervised learning. Recently the performance of these traditional classifiers has degraded as the number of documents has increased. This is because along with this growth in the number of documents has come an increase in the number of categories. This paper approaches this problem differently from cur…
Recent Advances in Hierarchical Multi-label Text Classification: A Survey
Beijing University Of Posts and Telecommunications（北京邮电大学）；Patent Examination Cooperation, CNIPA（国家知识产权局专利审查协作中心）；Zhongtukexin Co. Ltd.（中图科信有限公司）；Deakin University（迪肯大学）
Document classification is a challenging task with important applications. The deep learning approaches to the problem have gained much attention recently. Despite the progress, the proposed models do not incorporate the knowledge of the document structure in the architecture efficiently and not take into account the contexting importance of words and sentences. In this paper, we propose a new approach based on a combination of convolutional neural networks, gated recurrent units, and attention mechanisms for document classification tasks. The main contribution of this work is the use of convo…
Text classification is a primary task in natural language processing (NLP). Recently, graph neural networks (GNNs) have developed rapidly and been applied to text classification tasks. As a special kind of graph data, the tree has a simpler data structure and can provide rich hierarchical information for text classification. Inspired by the structural entropy, we construct the coding tree of the graph by minimizing the structural entropy and propose HINT, which aims to make full use of the hierarchical information contained in the text for the task of text classification. Specifically, we firs…
Beijing University of Posts and Telecommunications（北京邮电大学）
Hierarchical multi-label academic text classification (HMTC) is to assign academic texts into a hierarchically structured labeling system. We propose an attention-based hierarchical multi-label classification algorithm of academic texts (AHMCA) by integrating features such as text, keywords, and hierarchical structure, the academic documents are classified into the most relevant categories. We utilize word2vec and BiLSTM to obtain embedding and latent vector representations of text, keywords, and hierarchies. We use hierarchical attention mechanism to capture the associations between keywords,…
University of California, Santa Barbara（加州大学圣巴巴拉分校）
Existing work on automated hate speech detection typically focuses on binary classification or on differentiating among a small set of categories. In this paper, we propose a novel method on a fine-grained hate speech classification task, which focuses on differentiating among 40 hate groups of 13 different hate group categories. We first explore the Conditional Variational Autoencoder (CVAE) as a discriminative model and then extend it to a hierarchical architecture to utilize the additional hate category information for more accurate prediction. Experimentally, we show that incorporating the…
University of Virginia（弗吉尼亚大学）；University of California, Los Angeles (UCLA)（加利福尼亚大学洛杉矶分校）；The Aga Khan University（阿迦汗大学）；University of Zambia School of Medicine（赞比亚大学医学院）；Queen Mary University of London（伦敦玛丽女王大学）
Image classification is central to the big data revolution in medicine. Improved information processing methods for diagnosis and classification of digital medical images have shown to be successful via deep learning approaches. As this field is explored, there are limitations to the performance of traditional supervised classifiers. This paper outlines an approach that is different from the current medical image classification tasks that view the issue as multi-class classification. We performed a hierarchical classification using our Hierarchical Medical Image classification (HMIC) approach.…
North China Electric Power University（华北电力大学）；Institute of Software, Chinese Academy of Sciences（中国科学院软件研究所）
Many important classification problems in the real-world consist of a large number of closely related categories in a hierarchical structure or taxonomy. Hierarchical multi-label text classification (HMTC) with higher accuracy over large sets of closely related categories organized in a hierarchy or taxonomy has become a challenging problem. In this paper, we present a hierarchical and fine-tuning approach based on the Ordered Neural LSTM neural network, abbreviated as HFT-ONLSTM, for more accurate level-by-level HMTC. First, we present a novel approach to learning the joint embeddings based o…
Fuzzy Cognitive Maps (FCMs) are soft computing technique that follows an approach similar to human reasoning and human decision-making process, making them a valuable modeling and simulation methodology. Medical Decision Systems are complex systems consisting of many factors that may be complementary, contradictory, and competitive; these factors influence each other and determine the overall diagnosis with a different degree. Thus, FCMs are suitable to model Medical Decision Support Systems. The proposed work therefore uses FCMs arranged in hierarchical structure to classify between Diabetes,…
University of Warwick（华威大学）；King’s College London（伦敦国王学院）；Alan Turing Institute（阿兰·图灵研究所）
Recent years have witnessed increasing interests in developing interpretable models in Natural Language Processing (NLP). Most existing models aim at identifying input features such as words or phrases important for model predictions. Neural models developed in NLP however often compose word semantics in a hierarchical manner and text classification requires hierarchical modelling to aggregate local information in order to deal with topic and label shifts more effectively. As such, interpretation by words or phrases only cannot faithfully explain model decisions in text classification. This pa…
Due to the lack of structured knowledge applied in learning distributed representation of cate- gories, existing work cannot incorporate category hierarchies into entity information. We propose a framework that embeds entities and categories into a semantic space by integrating structured knowledge and taxonomy hierarchy from large knowledge bases. The framework allows to com- pute meaningful semantic relatedness between entities and categories. Our framework can han- dle both single-word concepts and multiple-word concepts with superior performance on concept categorization and yield state of…
Hi-LSplat: Hierarchical 3D Language Gaussian Splatting
College of Computer Science and Technology, Zhejiang University（浙江大学计算机科学与技术学院）；College of Biomedical Engineering and Instrument Science, Zhejiang University（浙江大学生物医学工程与仪器科学学院）；Zhejiang University-University of Illinois Urbana-Champaign Institute, Zhejiang University（浙江大学-伊利诺伊大学厄巴纳-香槟分校联合研究所）
Baidu Research, Baidu Inc.（百度研究院、百度公司）；University of Virginia（弗吉尼亚大学）；Tsinghua University（清华大学）
Short text classification is a fundamental task in natural language processing. It is hard due to the lack of context information and labeled data in practice. In this paper, we propose a new method called SHINE, which is based on graph neural network (GNN), for short text classification. First, we model the short text dataset as a hierarchical heterogeneous graph consisting of word-level component graphs which introduce more semantic and syntactic information. Then, we dynamically learn a short document graph that facilitates effective label propagation among similar short texts. Thus, compar…
Constraint-Aware Hierarchical Search for Regulation-Driven Fine-Grained Classification
Gusu Laboratory of Materials（姑苏材料实验室）；Chongqing Institute of Engineering（重庆工程学院）；Suzhou Digital China Wuxin Intelligent Technology Co., Ltd.（苏州神州数码五新智能科技有限公司）
University of Arizona（亚利桑那大学）；University of Birmingham（伯明翰大学）
Prior work has demonstrated that question classification (QC), recognizing the problem domain of a question, can help answer it more accurately. However, developing strong QC algorithms has been hindered by the limited size and complexity of annotated data available. To address this, we present the largest challenge dataset for QC, containing 7,787 science exam questions paired with detailed classification labels from a fine-grained hierarchical taxonomy of 406 problem domains. We then show that a BERT-based model trained on this dataset achieves a large (+0.12 MAP) gain compared with previous…
