Takagi-Sugeno-Kang (TSK) fuzzy systems are flexible and interpretable machine learning models; however, they may not be easily optimized when the data size is large, and/or the data dimensionality is high. This paper proposes a mini-batch gradient descent (MBGD) based algorithm to efficiently and effectively train TSK fuzzy classifiers. It integrates two novel techniques: 1) uniform regularization (UR), which forces the rules to have similar average contributions to the output, and hence to increase the generalization performance of the TSK classifier; and, 2) batch normalization (BN), which extends BN from deep neural networks to TSK fuzzy classifiers to expedite the convergence and improve the generalization performance. Experiments on 12 UCI datasets from various application domains, with varying size and dimensionality, demonstrated that UR and BN are effective individually, and integrating them can further improve the classification performance.
Huazhong University of Science and Technology（华中科技大学）
This paper presents PyTSK, a Python toolbox for developing Takagi-Sugeno-Kang (TSK) fuzzy systems. Based on scikit-learn and PyTorch, PyTSK allows users to optimize TSK fuzzy systems using fuzzy clustering or mini-batch gradient descent (MBGD) based algorithms. Several state-of-the-art MBGD-based optimization algorithms are implemented in the toolbox, which can improve the generalization performance of TSK fuzzy systems, especially for big data applications. PyTSK can also be easily extended and customized for more complicated algorithms, such as modifying the structure of TSK fuzzy systems, d…
Huazhong University of Science and Technology（华中科技大学）；School of Artificial Intelligence and Automation（人工智能与自动化学院）；Key Laboratory of the Ministry of Education for Image Processing and Intelligent Control（教育部图像处理与智能控制重点实验室）
To effectively train Takagi-Sugeno-Kang (TSK) fuzzy systems for regression problems, a Mini-Batch Gradient Descent with Regularization, DropRule, and AdaBound (MBGD-RDA) algorithm was recently proposed. It has demonstrated superior performances; however, there are also some limitations, e.g., it does not allow the user to specify the number of rules directly, and only Gaussian MFs can be used. This paper proposes two variants of MBGD-RDA to remedy these limitations, and show that they outperform the original MBGD-RDA and the classical ANFIS algorithms with the same number of rules. Furthermore…
Huazhong University of Science and Technology（华中科技大学）
Takagi-Sugeno-Kang (TSK) fuzzy systems are very useful machine learning models for regression problems. However, to our knowledge, there has not existed an efficient and effective training algorithm that ensures their generalization performance, and also enables them to deal with big data. Inspired by the connections between TSK fuzzy systems and neural networks, we extend three powerful neural network optimization techniques, i.e., mini-batch gradient descent, regularization, and AdaBound, to TSK fuzzy systems, and also propose three novel techniques (DropRule, DropMF, and DropMembership) spe…
Huazhong University of Science and Technology（华中科技大学）；Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）
To effectively optimize Takagi-Sugeno-Kang (TSK) fuzzy systems for regression problems, a mini-batch gradient descent with regularization, DropRule, and AdaBound (MBGD-RDA) algorithm was recently proposed. This paper further proposes FCM-RDpA, which improves MBGD-RDA by replacing the grid partition approach in rule initialization by fuzzy c-means clustering, and AdaBound by Powerball AdaBelief, which integrates recently proposed Powerball gradient and AdaBelief to further expedite and stabilize parameter optimization. Extensive experiments on 22 regression datasets with various sizes and dimen…
NEC Laboratories Europe（NEC欧洲实验室）；Paderborn University（帕德博恩大学）
The problem of adaptive learning from evolving and possibly non-stationary data streams has attracted a lot of interest in machine learning in the recent past, and also stimulated research in related fields, such as computational intelligence and fuzzy systems. In particular, several rule-based methods for the incremental induction of regression models have been proposed. In this paper, we develop a method that combines the strengths of two existing approaches rooted in different learning paradigms. More concretely, our method adopts basic principles of the state-of-the-art learning algorithm …
Huazhong University of Science and Technology（华中科技大学）；Shandong University of Finance and Economics（山东财经大学）
Fuzzy c-means based clustering algorithms are frequently used for Takagi-Sugeno-Kang (TSK) fuzzy classifier antecedent parameter estimation. One rule is initialized from each cluster. However, most of these clustering algorithms are unsupervised, which waste valuable label information in the training data. This paper proposes a supervised enhanced soft subspace clustering (SESSC) algorithm, which considers simultaneously the within-cluster compactness, between-cluster separation, and label information in clustering. It can effectively deal with high-dimensional data, be used as a classifier al…
Huzhou University（湖州师范学院）；Zhejiang Normal University（浙江师范大学）；Jiangnan University（江南大学）；Huazhong University of Science and Technology（华中科技大学）；Hangzhou Dianzi University（杭州电子科技大学）；Zhejiang University（浙江大学）
High-order Takagi-Sugeno-Kang (TSK) fuzzy classifiers possess powerful classification performance yet have fewer fuzzy rules, but always be impaired by its exponential growth training time and poorer interpretability owing to High-order polynomial used in consequent part of fuzzy rule, while Low-order TSK fuzzy classifiers run quickly with high interpretability, however they usually require more fuzzy rules and perform relatively not very well. Address this issue, a novel TSK fuzzy classifier embeded with knowledge distillation in deep learning called HTSK-LLM-DKD is proposed in this study. HT…
Jiangnan University（江南大学）；The Hong Kong Polytechnic University（香港理工大学）
The superior interpretability and uncertainty modeling ability of Takagi-Sugeno-Kang fuzzy system (TSK FS) make it possible to describe complex nonlinear systems intuitively and efficiently. However, classical TSK FS usually adopts the whole feature space of the data for model construction, which can result in lengthy rules for high-dimensional data and lead to degeneration in interpretability. Furthermore, for highly nonlinear modeling task, it is usually necessary to use a large number of rules which further weakens the clarity and interpretability of TSK FS. To address these issues, a conci…
Huazhong University of Science and Technology（华中科技大学）
Takagi-Sugeno-Kang (TSK) fuzzy system with Gaussian membership functions (MFs) is one of the most widely used fuzzy systems in machine learning. However, it usually has difficulty handling high-dimensional datasets. This paper explores why TSK fuzzy systems with Gaussian MFs may fail on high-dimensional inputs. After transforming defuzzification to an equivalent form of softmax function, we find that the poor performance is due to the saturation of softmax. We show that two defuzzification operations, LogTSK and HTSK, the latter of which is first proposed in this paper, can avoid the saturatio…
Institute of Microelectronics, Chinese Academy of Sciences（中国科学院微电子研究所）；University of Chinese Academy of Sciences（中国科学院大学）；Durham University（杜伦大学）；Teesside University（提赛德大学）
Situating at the core of Artificial Intelligence (AI), Machine Learning (ML), and more specifically, Deep Learning (DL) have embraced great success in the past two decades. However, unseen class label prediction is far less explored due to missing classes being invisible in training ML or DL models. In this work, we propose a fuzzy inference system to cope with such a challenge by adopting TSK+ fuzzy inference engine in conjunction with the Curvature-based Feature Selection (CFS) method. The practical feasibility of our system has been evaluated by predicting the positioning labels of networki…
Data mining techniques can be used to discover useful patterns by exploring and analyzing data and it's feasible to synergitically combine machine learning tools to discover fuzzy classification rules.In this paper, an adaptive Neuro fuzzy network with TSK fuzzy type and an improved quantum subtractive clustering has been developed. Quantum clustering (QC) is an intuition from quantum mechanics which uses Schrodinger potential and time-consuming gradient descent method. The principle advantage and shortcoming of QC is analyzed and based on its shortcomings, an improved algorithm through a subt…
School of Wangzheng Microelectronics, Changzhou University（常州大学王正微电子学院）；School of Electrical Engineering and Automation, Changshu Institute of Technology（常熟理工学院电气与自动化工程学院）
Huazhong University of Science and Technology（华中科技大学）；University of Technology, Sydney（悉尼科技大学）
Fuzzy systems have achieved great success in numerous applications. However, there are still many challenges in designing an optimal fuzzy system, e.g., how to efficiently optimize its parameters, how to balance the trade-off between cooperations and competitions among the rules, how to overcome the curse of dimensionality, how to increase its generalization ability, etc. Literature has shown that by making appropriate connections between fuzzy systems and other machine learning approaches, good practices from other domains may be used to improve the fuzzy systems, and vice versa. This paper g…
Transfer learning can address the learning tasks of unlabeled data in the target domain by leveraging plenty of labeled data from a different but related source domain. A core issue in transfer learning is to learn a shared feature space in where the distributions of the data from two domains are matched. This learning process can be named as transfer representation learning (TRL). The feature transformation methods are crucial to ensure the success of TRL. The most commonly used feature transformation method in TRL is kernel-based nonlinear mapping to the high-dimensional space followed by li…
Centro de Investigación en Tecnoloxías da Información (CITIUS)（信息技术研究中心（CITIUS））；Universidade de Santiago de Compostela（圣地亚哥德孔波斯特拉大学）
In regression problems, the use of TSK fuzzy systems is widely extended due to the precision of the obtained models. Moreover, the use of simple linear TSK models is a good choice in many real problems due to the easy understanding of the relationship between the output and input variables. In this paper we present FRULER, a new genetic fuzzy system for automatically learning accurate and simple linguistic TSK fuzzy rule bases for regression problems. In order to reduce the complexity of the learned models while keeping a high accuracy, the algorithm consists of three stages: instance selectio…
Shahid Beheshti University（沙希德·贝赫什提大学）；Amirkabir University of Technology（阿米尔卡比尔理工大学）
In this paper, a novel stepwise learning approach based on estimating desired premise parts' outputs by solving a constrained optimization problem is proposed. This learning approach does not require backpropagating the output error to learn the premise parts' parameters. Instead, the near best output values of the rules premise parts are estimated and their parameters are changed to reduce the error between current premise parts' outputs and the estimated desired ones. Therefore, the proposed learning method avoids error backpropagation, which lead to vanishing gradient and consequently getti…
A Novel TSK Fuzzy System Incorporating Multi-view Collaborative Transfer Learning for Personalized Epileptic EEG Detection
School of Artificial Intelligence and Computer Science, Jiangnan University and Engineering Research Center of Intelligent Technology for Healthcare, Ministry of Education（江南大学人工智能与计算机科学学院，智能医疗技术工程研究中心，教育部）；School of Computer and Information Engineering, Institute for Artificial Intelligence, Shanghai Polytechnic University（上海理工大学计算机与信息工程学院，人工智能研究院）；School of Internet of Things Engineering, Wuxi University（无锡大学物联网工程学院）
Institute for Infocomm Research, A*STAR（新加坡科技研究局资讯通信研究院）；Nanyang Technological University（南洋理工大学）
Multi-view data clustering refers to categorizing a data set by making good use of related information from multiple representations of the data. It becomes important nowadays because more and more data can be collected in a variety of ways, in different settings and from different sources, so each data set can be represented by different sets of features to form different views of it. Many approaches have been proposed to improve clustering performance by exploring and integrating heterogeneous information underlying different views. In this paper, we propose a new multi-view fuzzy clustering…
Institute for Infocomm Research, A*STAR（新加坡科技研究局资讯通信研究院）；Nanyang Technological University（南洋理工大学）
Incremental clustering approaches have been proposed for handling large data when given data set is too large to be stored. The key idea of these approaches is to find representatives to represent each cluster in each data chunk and final data analysis is carried out based on those identified representatives from all the chunks. However, most of the incremental approaches are used for single view data. As large multi-view data generated from multiple sources becomes prevalent nowadays, there is a need for incremental clustering approaches to handle both large and multi-view data. In this paper…
Xsensor Technology Corp（Xsensor科技公司）；University of Alberta（阿尔伯塔大学）
The combination of neural network and fuzzy systems into neuro-fuzzy systems integrates fuzzy reasoning rules into the connectionist networks. However, the existing neuro-fuzzy systems are developed under shallow structures having lower generalization capacity. We propose the first end-to-end deep neuro-fuzzy network and investigate its application for image classification. Two new operations are developed based on definitions of Takagi-Sugeno-Kang (TSK) fuzzy model namely fuzzy inference operation and fuzzy pooling operations; stacks of these operations comprise the layers in this network. We…
China University of Petroleum (East China)（中国石油大学（华东））；Indian Statistical Institute（印度统计研究所）
A major limitation of fuzzy or neuro-fuzzy systems is their failure to deal with high-dimensional datasets. This happens primarily due to the use of T-norm, particularly, product or minimum (or a softer version of it). Thus, there are hardly any work dealing with datasets with dimensions more than hundred or so. Here, we propose a neuro-fuzzy framework that can handle datasets with dimensions even more than 7000! In this context, we propose an adaptive softmin (Ada-softmin) which effectively overcomes the drawbacks of ``numeric underflow" and ``fake minimum" that arise for existing fuzzy syste…
On-board Sonar Data Classification for Path Following in Underwater Vehicles using Fast Interval Type-2 Fuzzy Extreme Learning Machine
Hamlyn Centre for Robotic Surgery, Imperial College London（机器人手术哈明中心，帝国理工学院伦敦分校）；Center for Engineering and Industrial Development (CIDESI)（工程与工业发展中心（CIDESI））
The University of Suwon（水原大学）；Linyi University（临沂大学）；University of Alberta（阿尔伯塔大学）；King Abdulaziz University（阿卜杜勒阿齐兹国王大学）；Systems Research Institute, Polish Academy of Sciences（波兰科学院系统研究所）；Beijing Normal University（北京师范大学）
In the conventional Takagi-Sugeno-Kang (TSK)-type fuzzy models, constant or linear functions are usually utilized as the consequent parts of the fuzzy rules, but they cannot effectively describe the behavior within local regions defined by the antecedent parts. In this article, a theoretical and practical design methodology is developed to address this problem. First, the information granulation (Fuzzy C-Means) method is applied to capture the structure in the data and split the input space into subspaces, as well as form the antecedent parts. Second, the quadratic polynomials (QPs) are employ…
DRIMV_TSK: An Interpretable Surgical Evaluation Model for Incomplete Multi-View Rectal Cancer Data
Nantong University（南通大学）；Jiangnan University（江南大学）；Affiliated Hospital of Jiangnan University（江南大学附属医院）；University of Nottingham（诺丁汉大学）；Hong Kong Polytechnic University（香港理工大学）
Imperial College London（伦敦帝国学院）；Sciemus Ltd（Sciemus有限公司）；University of Oxford（牛津大学）
In a recent paper [1] we introduced the Fuzzy Bayesian Learning (FBL) paradigm where expert opinions can be encoded in the form of fuzzy rule bases and the hyper-parameters of the fuzzy sets can be learned from data using a Bayesian approach. The present paper extends this work for selecting the most appropriate rule base among a set of competing alternatives, which best explains the data, by calculating the model evidence or marginal likelihood. We explain why this is an attractive alternative over simply minimizing a mean squared error metric of prediction and show the validity of the propos…
The Vapnik-Chervonenkis (VC) dimension measures the complexity of a learning machine, and a low VC dimension leads to good generalization. The recently proposed Minimal Complexity Machine (MCM) learns a hyperplane classifier by minimizing an exact bound on the VC dimension. This paper extends the MCM classifier to the fuzzy domain. The use of a fuzzy membership is known to reduce the effect of outliers, and to reduce the effect of noise on learning. Experimental results show, that on a number of benchmark datasets, the the fuzzy MCM classifier outperforms SVMs and the conventional MCM in terms…
Methods for analyzing or learning from "fuzzy data" have attracted increasing attention in recent years. In many cases, however, existing methods (for precise, non-fuzzy data) are extended to the fuzzy case in an ad-hoc manner, and without carefully considering the interpretation of a fuzzy set when being used for modeling data. Distinguishing between an ontic and an epistemic interpretation of fuzzy set-valued data, and focusing on the latter, we argue that a "fuzzification" of learning algorithms based on an application of the generic extension principle is not appropriate. In fact, the exte…
Tel Aviv University（特拉维夫大学）；University of California, San Diego（加利福尼亚大学圣迭戈分校）；University of Massachusetts Amherst（马萨诸塞大学阿默斯特分校）
The fuzzy or soft $k$-means objective is a popular generalization of the well-known $k$-means problem, extending the clustering capability of the $k$-means to datasets that are uncertain, vague, and otherwise hard to cluster. In this paper, we propose a semi-supervised active clustering framework, where the learner is allowed to interact with an oracle (domain expert), asking for the similarity between a certain set of chosen items. We study the query and computational complexities of clustering in this framework. We prove that having a few of such similarity queries enables one to get a polyn…
An evidential time-to-event prediction model based on Gaussian random fuzzy numbers
Saw Swee Hock School of Public Health, National University of Singapore（新加坡国立大学苏瑞福公共卫生学院）；Université de technologie de Compiègne（贡比涅技术大学）；CNRS（法国国家科学研究中心）；Heudiasyc（于迪阿斯实验室）；Institut Universitaire de France（法国大学研究院）；Institute of Data Science, National University of Singapore（新加坡国立大学数据科学研究院）
The existence of large volumes of time series data in many applications has motivated data miners to investigate specialized methods for mining time series data. Clustering is a popular data mining method due to its powerful exploratory nature and its usefulness as a preprocessing step for other data mining techniques. This article develops two novel clustering algorithms for time series data that are extensions of a crisp c-shapes algorithm. The two new algorithms are heuristic derivatives of fuzzy c-means (FCM). Fuzzy c-Shapes plus (FCS+) replaces the inner product norm in the FCM model with…
Text Classification is a challenging and a red hot field in the current scenario and has great importance in text categorization applications. A lot of research work has been done in this field but there is a need to categorize a collection of text documents into mutually exclusive categories by extracting the concepts or features using supervised learning paradigm and different classification algorithms. In this paper, a new Fuzzy Similarity Based Concept Mining Model (FSCMM) is proposed to classify a set of text documents into pre - defined Category Groups (CG) by providing them training and…
Universidade Federal de Pernambuco（伯南布哥联邦大学）；Centro de Informática（信息中心）
The clustering methods have been used in a variety of fields such as image processing, data mining, pattern recognition, and statistical analysis. Generally, the clustering algorithms consider all variables equally relevant or not correlated for the clustering task. Nevertheless, in real situations, some variables can be correlated or may be more or less relevant or even irrelevant for this task. This paper proposes partitioning fuzzy clustering algorithms based on Euclidean, City-block and Mahalanobis distances and entropy regularization. These methods are an iterative three steps algorithms …
Islamic Azad University（伊斯兰阿扎德大学）；K. N. Toosi University of Technology（K. N. 图西理工大学）；Deakin University（迪肯大学）
Many works have been done to handle the uncertainties in the data using type 1 fuzzy regression. Few type 2 fuzzy regression works used interval type 2 for indeterminate modeling using type 1 fuzzy membership. The current survey proposes a triangular type-2 fuzzy regression (TT2FR) model to ameliorate the efficiency of the model by handling the uncertainty in the data. The triangular secondary membership function is used instead of widely used interval type models. In the proposed model, vagueness in primary and secondary fuzzy sets is minimized and also, a specified x-plane of observed value …
In this study, a new Stacked Generalization technique called Fuzzy Stacked Generalization (FSG) is proposed to minimize the difference between N -sample and large-sample classification error of the Nearest Neighbor classifier. The proposed FSG employs a new hierarchical distance learning strategy to minimize the error difference. For this purpose, we first construct an ensemble of base-layer fuzzy k- Nearest Neighbor (k-NN) classifiers, each of which receives a different feature set extracted from the same sample set. The fuzzy membership values computed at the decision space of each fuzzy k-N…
McGill University（麦吉尔大学）；Montreal Institute for Learning Algorithms (MILA)（蒙特利尔学习算法研究所）；Institut Pasteur（巴斯德研究所）；Mines ParisTech（巴黎高等矿业学院）；Paris-Sciences-et-Lettres Research University（巴黎文理研究大学）
RNA 3D motifs are recurrent substructures, modelled as networks of base pair interactions, which are crucial for understanding structure-function relationships. The task of automatically identifying such motifs is computationally hard, and remains a key challenge in the field of RNA structural biology and network analysis. State of the art methods solve special cases of the motif problem by constraining the structural variability in occurrences of a motif, and narrowing the substructure search space. Here, we relax these constraints by posing the motif finding problem as a graph representation…
