Despite the success of deep learning in computer vision and natural language processing, Gradient Boosted Decision Tree (GBDT) is yet one of the most powerful tools for applications with tabular data such as e-commerce and FinTech. However, applying GBDT to multi-task learning is still a challenge. Unlike deep models that can jointly learn a shared latent representation across multiple tasks, GBDT can hardly learn a shared tree structure. In this paper, we propose Multi-task Gradient Boosting Machine (MT-GBM), a GBDT-based method for multi-task learning. The MT-GBM can find the shared tree structures and split branches according to multi-task losses. First, it assigns multiple outputs to each leaf node. Next, it computes the gradient corresponding to each output (task). Then, we also propose an algorithm to combine the gradients of all tasks and update the tree. Finally, we apply MT-GBM to LightGBM. Experiments show that our MT-GBM improves the performance of the main task significantly, which means the proposed MT-GBM is efficient and effective.
Google（谷歌）；University of Chicago（芝加哥大学）；EPFL（洛桑联邦理工学院）
Gradient Boosting Machine (GBM) is an extremely powerful supervised learning algorithm that is widely used in practice. GBM routinely features as a leading algorithm in machine learning competitions such as Kaggle and the KDDCup. In this work, we propose Accelerated Gradient Boosting Machine (AGBM) by incorporating Nesterov's acceleration techniques into the design of GBM. The difficulty in accelerating GBM lies in the fact that weak (inexact) learners are commonly used, and therefore the errors can accumulate in the momentum term. To overcome it, we design a "corrected pseudo residual" and fi…
Peter the Great St.Petersburg Polytechnic University（圣彼得堡彼得大帝理工大学）
A method for the local and global interpretation of a black-box model on the basis of the well-known generalized additive models is proposed. It can be viewed as an extension or a modification of the algorithm using the neural additive model. The method is based on using an ensemble of gradient boosting machines (GBMs) such that each GBM is learned on a single feature and produces a shape function of the feature. The ensemble is composed as a weighted sum of separate GBMs resulting a weighted sum of shape functions which form the generalized additive model. GBMs are built in parallel using ran…
Gradient Boosting Machines (GBM) are among the go-to algorithms on tabular data, which produce state of the art results in many prediction tasks. Despite its popularity, the GBM framework suffers from a fundamental flaw in its base learners. Specifically, most implementations utilize decision trees that are typically biased towards categorical variables with large cardinalities. The effect of this bias was extensively studied over the years, mostly in terms of predictive performance. In this work, we extend the scope and study the effect of biased base learners on GBM feature importance (FI) m…
Peter the Great St.Petersburg Polytechnic University（彼得大帝圣彼得堡理工大学）
A new attention-based model for the gradient boosting machine (GBM) called AGBoost (the attention-based gradient boosting) is proposed for solving regression problems. The main idea behind the proposed AGBoost model is to assign attention weights with trainable parameters to iterations of GBM under condition that decision trees are base learners in GBM. Attention weights are determined by applying properties of decision trees and by using the Huber's contamination model which provides an interesting linear dependence between trainable parameters of the attention and the attention weights. This…
Booth School of Business, University of Chicago（芝加哥大学布斯商学院）；MIT Sloan School of Management（麻省理工学院斯隆管理学院）；Operations Research Center（运筹学中心）；MIT Center for Statistics（麻省理工学院统计中心）
Gradient Boosting Machine (GBM) introduced by Friedman is a powerful supervised learning algorithm that is very widely used in practice---it routinely features as a leading algorithm in machine learning competitions such as Kaggle and the KDDCup. In spite of the usefulness of GBM in practice, our current theoretical understanding of this method is rather limited. In this work, we propose Randomized Gradient Boosting Machine (RGBM) which leads to substantial computational gains compared to GBM, by using a randomization scheme to reduce search in the space of weak-learners. We derive novel compu…
University of Oxford（牛津大学）；University of Strathclyde（思克莱德大学）
Artificial intelligence (AI) and machine learning (ML) have become vital to remain competitive for financial services companies around the globe. The two models currently competing for the pole position in credit risk management are deep learning (DL) and gradient boosting machines (GBM). This paper benchmarked those two algorithms in the context of credit scoring using three distinct datasets with different features to account for the reality that model choice/power is often dependent on the underlying characteristics of the dataset. The experiment has shown that GBM tends to be more powerful…
Convolutional neural networks (CNNs) and transformers, which are composed of multiple processing layers and blocks to learn the representations of data with multiple abstract levels, are the most successful machine learning models in recent years. However, millions of parameters and many blocks make them difficult to be trained, and sometimes several days or weeks are required to find an ideal architecture or tune the parameters. Within this paper, we propose a new algorithm for boosting Deep Convolutional Neural Networks (BoostCNN) to combine the merits of dynamic feature selection and BoostC…
Use of Boosting Algorithms in Household-Level Poverty Measurement: A Machine Learning Approach to Predict and Classify Household Wealth Quintiles in the Philippines
Amherst College（阿默斯特学院）；De La Salle University Integrated School Manila（德拉萨大学马尼拉附属中学）
This article provides an overview of Supervised Machine Learning (SML) with a focus on applications to banking. The SML techniques covered include Bagging (Random Forest or RF), Boosting (Gradient Boosting Machine or GBM) and Neural Networks (NNs). We begin with an introduction to ML tasks and techniques. This is followed by a description of: i) tree-based ensemble algorithms including Bagging with RF and Boosting with GBMs, ii) Feedforward NNs, iii) a discussion of hyper-parameter optimization techniques, and iv) machine learning interpretability. The paper concludes with a comparison of the …
Gradient Boosting Machines (GBM) are hugely popular for solving tabular data problems. However, practitioners are not only interested in point predictions, but also in probabilistic predictions in order to quantify the uncertainty of the predictions. Creating such probabilistic predictions is difficult with existing GBM-based solutions: they either require training multiple models or they become too computationally expensive to be useful for large-scale settings. We propose Probabilistic Gradient Boosting Machines (PGBM), a method to create probabilistic predictions with a single ensemble of d…
University of Tuebingen（图宾根大学）；Amazon（亚马逊）；Deekard（迪卡德）
Gradient boosting machines (GBMs) based on decision trees consistently demonstrate state-of-the-art results on regression and classification tasks with tabular data, often outperforming deep neural networks. However, these models do not provide well-calibrated predictive uncertainties, which prevents their use for decision making in high-risk applications. The Bayesian treatment is known to improve predictive uncertainty calibration, but previously proposed Bayesian GBM methods are either computationally expensive, or resort to crude approximations. Variational inference is often used to imple…
Baidu Inc.（百度公司）；Beijing University of Posts and Telecommunications（北京邮电大学）；Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences（中国科学院深圳先进技术研究院）
Federated machine learning systems have been widely used to facilitate the joint data analytics across the distributed datasets owned by the different parties that do not trust each others. In this paper, we proposed a novel Gradient Boosting Machines (GBM) framework SecureGBM built-up with a multi-party computation model based on semi-homomorphic encryption, where every involved party can jointly obtain a shared Gradient Boosting machines model while protecting their own data from the potential privacy leakage and inferential identification. More specific, our work focused on a specific "dual…
Inductive inference of gradient-boosted decision trees on graphs for insurance fraud detection
Data Office, Allianz Benelux（Allianz Benelux 数据办公室）；KU Leuven University of Antwerp-imec（安特卫普-imec 联合大学KU Leuven）；KU Leuven；University of Antwerp-imec（安特卫普-imec 大学）
Current implementations of Gradient Boosting Machines are mostly designed for single-target regression tasks and commonly assume independence between responses when used in multivariate settings. As such, these models are not well suited if non-negligible dependencies exist between targets. To overcome this limitation, we present an extension of XGBoostLSS that models multiple targets and their dependencies in a probabilistic regression setting. Empirical results show that our approach outperforms existing GBMs with respect to runtime and compares well in terms of accuracy.
Sinovation Ventures AI Institute（创新工场人工智能研究院）；Baiont Technology（百融科技）；Nanjing University（南京大学）
Gradient Boosting Machine has proven to be one successful function approximator and has been widely used in a variety of areas. However, since the training procedure of each base learner has to take the sequential order, it is infeasible to parallelize the training process among base learners for speed-up. In addition, under online or incremental learning settings, GBMs achieved sub-optimal performance due to the fact that the previously trained base learners can not adapt with the environment once trained. In this work, we propose the soft Gradient Boosting Machine (sGBM) by wiring multiple d…
Development and Comparative Analysis of Machine Learning Models for Hypoxemia Severity Triage in CBRNE Emergency Scenarios Using Physiological and Demographic Data from Medical-Grade Devices
SADC-CDSS IA PEDIATRICS, CHU Sainte-Justine, Montreal, Canada（SADC-CDSS IA儿科，圣-朱斯特医院，蒙特利尔，加拿大）；Solutions Applicare AI Inc., Montreal, Canada（应用爱智AI公司，蒙特利尔，加拿大）；Université de Montréal, Canada（蒙特利尔大学，加拿大）；MEDINT CBRNE Group, Montreal, Canada（MEDINT CBRNE组，蒙特利尔，加拿大）
Although many methods have been proposed to enhance the transferability of adversarial perturbations, these methods are designed in a heuristic manner, and the essential mechanism for improving adversarial transferability is still unclear. This paper summarizes the common mechanism shared by twelve previous transferability-boosting methods in a unified view, i.e., these methods all reduce game-theoretic interactions between regional adversarial perturbations. To this end, we focus on the attacking utility of all interactions between regional adversarial perturbations, and we first discover and…
Technology and collaboration enable dramatic increases in the size of psychological and psychiatric data collections, but finding structure in these large data sets with many collected variables is challenging. Decision tree ensembles like random forests (Strobl, Malley, and Tutz, 2009) are a useful tool for finding structure, but are difficult to interpret with multiple outcome variables which are often of interest in psychology. To find and interpret structure in data sets with multiple outcomes and many predictors (possibly exceeding the sample size), we introduce a multivariate extension t…
Boosting methods combine a set of moderately accurate weaklearners to form a highly accurate predictor. Despite the practical importance of multi-class boosting, it has received far less attention than its binary counterpart. In this work, we propose a fully-corrective multi-class boosting formulation which directly solves the multi-class problem without dividing it into multiple binary classification problems. In contrast, most previous multi-class boosting algorithms decompose a multi-boost problem into multiple binary boosting problems. By explicitly deriving the Lagrange dual of the primal…
Enhancing Credit Default Prediction Using Boruta Feature Selection and DBSCAN Algorithm with Different Resampling Techniques
Department of Statistics, Western Michigan University（统计系，西部密歇根大学）；Department of Biostatistics and Data Science, Celia Scott Weatherhead School of Public Health and Tropical Medicine, Tulane University（生物统计学与数据科学系，Celia Scott Weatherhead公共卫生与热带医学学院，路易斯安那大学塔金斯分校）；School of Mathematical and Statistical Science, College of Sciences, University of Texas Rio Grande Valley（数学与统计科学学院，德克萨斯里奥格兰德谷大学）；Department of Economics, Western Michigan University（经济学系，西部密歇根大学）；Department of Economics, University of Ottawa（经济学系，渥太华大学）
Multi-Modal Machine Learning Framework for Predicting Early Recurrence of Brain Tumors Using MRI and Clinical Biomarkers
Department of Encephalopathy, Chengdu Pidu District Hospital of Traditional Chinese Medicine（脑病科，成都_pidu区中医医院）；Department of Tuina, Chengdu Pidu District Hospital of Traditional Chinese Medicine（推拿科，成都_pidu区中医医院）；Wuhan Hospital of Integrated Traditional Chinese and Western Medicine, Affiliated to Hubei University of Chinese Medicine（中西医结合医院，湖北中医药大学附属医院）；Department of Electrical and Computer Engineering, Carnegie Mellon University（电气与计算机工程系，卡内基梅隆大学）
Legible Inference（清晰推理公司）；University of Bristol（布里斯托大学）
The best-performing models in ML are not interpretable. If we can explain why they outperform, we may be able to replicate these mechanisms and obtain both interpretability and performance. One example are decision trees and their descendent gradient boosting machines (GBMs). These perform well in the presence of complex interactions, with tree depth governing the order of interactions. However, interactions cannot fully account for the depth of trees found in practice. We confirm 5 alternative hypotheses about the role of tree depth in performance in the absence of true interactions, and pres…
A Multi-Center Study on the Adaptability of a Shared Foundation Model for Electronic Health Records
Program in Child Health Evaluative Sciences, The Hospital for Sick Children, Toronto, ON, 1 Canada（儿童健康评估科学计划，多伦多医院儿童医学中心，多伦多，加拿大）；Stanford Center for Biomedical Informatics Research, Stanford University, Palo Alto, CA, USA（斯坦福大学生物医学信息学研究中心，斯坦福大学，帕洛阿尔托，美国）；Universidad del Norte, Barranquilla, Columbia（北方大学，巴兰基利亚，哥伦比亚）；Division of Haematology/Oncology, The Hospital for Sick Children, Toronto, ON, Canada（血液肿瘤科，多伦多医院儿童医学中心，多伦多，加拿大）
In many applications of supervised learning, multiple classification or regression outputs have to be predicted jointly. We consider several extensions of gradient boosting to address such problems. We first propose a straightforward adaptation of gradient boosting exploiting multiple output regression trees as base learners. We then argue that this method is only expected to be optimal when the outputs are fully correlated, as it forces the partitioning induced by the tree base learners to be shared by all outputs. We then propose a novel extension of gradient tree boosting to specifically ad…
Isomorphic Functionalities between Ant Colony and Ensemble Learning: Part II-On the Strength of Weak Learnability and the Boosting Paradigm
School of Mathematics and Statistics, Rochester Institute of Technology（罗彻斯特理工学院数学与统计学院）；Gosnell School of Life Sciences, Rochester Institute of Technology（罗彻斯特理工学院戈斯内尔生命科学学院）；Center for Imaging Science, Rochester Institute of Technology（罗彻斯特理工学院成像科学中心）
Multi-task learning (MTL) aims to make full use of the knowledge contained in multi-task supervision signals to improve the overall performance. How to make the knowledge of multiple tasks shared appropriately is an open problem for MTL. Most existing deep MTL models are based on parameter sharing. However, suitable sharing mechanism is hard to design as the relationship among tasks is complicated. In this paper, we propose a general framework called Multi-Task Neural Architecture Search (MTNAS) to efficiently find a suitable sharing route for a given MTL problem. MTNAS modularizes the sharing…
College of Artificial Intelligence, Nankai University（南开大学人工智能学院）
Grammar error correction (GEC) is an important application aspect of natural language processing techniques. The past decade has witnessed significant progress achieved in GEC for the sake of increasing popularity of machine learning and deep learning, especially in late 2010s when near human-level GEC systems are available. However, there is no prior work focusing on the whole recapitulation of the progress. We present the first survey in GEC for a comprehensive retrospect of the literature in this area. We first give the introduction of five public datasets, data annotation schema, two impor…
Survival modeling using deep learning, machine learning and statistical methods: A comparative analysis for predicting mortality after hospital admission
Duke-NUS Medical School（杜克-新加坡国立大学医学院）；Sorbonne University（索邦大学）；Stanford University（斯坦福大学）；Singapore General Hospital（新加坡中央医院）；National University of Singapore（新加坡国立大学）
Duke University（杜克大学）；Infinia ML, Inc（Infinia ML公司）；Microsoft Dynamics 365 AI Research（微软Dynamics 365人工智能研究院）；Facebook, Inc（脸书公司）
We propose a novel graph-driven generative model, that unifies multiple heterogeneous learning tasks into the same framework. The proposed model is based on the fact that heterogeneous learning tasks, which correspond to different generative processes, often rely on data with a shared graph structure. Accordingly, our model combines a graph convolutional network (GCN) with multiple variational autoencoders, thus embedding the nodes of the graph i.e., samples for the tasks) in a uniform manner while specializing their organization and usage to different tasks. With a focus on healthcare applica…
IoT-Based Environmental Control System for Fish Farms with Sensor Integration and Machine Learning Decision Support
Vel Tech Rangarajan Dr. Sagunthala R&D Institute of Science and Technology（维尔泰克拉詹德拉·萨贡塔拉科学技术研发学院）；Madanapalle Institute of Technology & Science（马达纳帕莱技术与科学学院）；Panimalar Engineering College（帕尼马勒工程学院）；R.M.K College of Engineering and Technology（R.M.K工程技术学院）
Trees to Flows and Back: Unifying Decision Trees and Diffusion Models
School of Computation, Information and Technology, Technical University of Munich, Germany（慕尼黑技术大学计算、信息与技术学院，德国）；Munich center for machine learning (MCML)（慕尼黑机器学习中心（MCML））
