Modern machine learning has achieved impressive prediction performance, but often sacrifices interpretability, a critical consideration in high-stakes domains such as medicine. In such settings, practitioners often use highly interpretable decision tree models, but these suffer from inductive bias against additive structure. To overcome this bias, we propose Fast Interpretable Greedy-Tree Sums (FIGS), which generalizes the CART algorithm to simultaneously grow a flexible number of trees in summation. By combining logical rules with addition, FIGS is able to adapt to additive structure while remaining highly interpretable. Extensive experiments on real-world datasets show that FIGS achieves state-of-the-art prediction performance. To demonstrate the usefulness of FIGS in high-stakes domains, we adapt FIGS to learn clinical decision instruments (CDIs), which are tools for guiding clinical decision-making. Specifically, we introduce a variant of FIGS known as G-FIGS that accounts for the heterogeneity in medical data. G-FIGS derives CDIs that reflect domain knowledge and enjoy improved specificity (by up to 20% over CART) without sacrificing sensitivity or interpretability. To provide further insight into FIGS, we prove that FIGS learns components of additive models, a property we refer to as disentanglement. Further, we show (under oracle conditions) that unconstrained tree-sum models leverage disentanglement to generalize more efficiently than single decision tree models when fitted to additive regression functions. Finally, to avoid overfitting with an unconstrained number of splits, we develop Bagging-FIGS, an ensemble version of FIGS that borrows the variance reduction techniques of random forests. Bagging-FIGS enjoys competitive performance with random forests and XGBoost on real-world datasets.
Multilayer networks have seen a resurgence under the umbrella of deep learning. Current deep learning algorithms train the layers of the network sequentially, improving algorithmic performance as well as providing some regularization. We present a new training algorithm for deep networks which trains \emph{each node in the network} sequentially. Our algorithm is orders of magnitude faster, creates more interpretable internal representations at the node level, while not sacrificing on the ultimate out-of-sample performance.
Rule ensembles are designed to provide a useful trade-off between predictive accuracy and model interpretability. However, the myopic and random search components of current rule ensemble methods can compromise this goal: they often need more rules than necessary to reach a certain accuracy level or can even outright fail to accurately model a distribution that can actually be described well with a few rules. Here, we present a novel approach aiming to fit rule ensembles of maximal predictive power for a given ensemble size (and thus model comprehensibility). In particular, we present an effic…
School of Computer Science（计算机科学学院）；Electronic Engineering University of Essex Colchester, United Kingdom（埃塞克斯大学电子工程学院伦敦郡，英国）；Department of Mathematics Slovak Academy of Sciences Bratislava, Slovakia（数学系斯洛伐克科学院布拉迪斯拉发，斯洛伐克）
CLARITree: Cholesky and Lookahead Accelerations for Regression with Interpretable Piecewise Linear Trees
Department of Computer Science, Duke University, Durham, USA（杜克大学计算机科学系，美国达勒姆）；Department of Computer Science, University of British Columbia, Vancouver, Canada（不列颠哥伦比亚大学计算机科学系，加拿大温哥华）
Interpretable clustering via optimal multi-way decision trees
Graduate School of Science and Technology, University of Tsukuba（科学技术研究生院，东京大学）；Institute of Systems and Information Engineering, University of Tsukuba（系统与信息工程研究所，东京大学）
INSEAD（欧洲工商管理学院）；Anderson School of Management, University of California, Los Angeles（加州大学洛杉矶分校安德森管理学院）
Optimal stopping is the problem of deciding when to stop a stochastic system to obtain the greatest reward, arising in numerous application areas such as finance, healthcare and marketing. State-of-the-art methods for high-dimensional optimal stopping involve approximating the value function or the continuation value, and then using that approximation within a greedy policy. Although such policies can perform very well, they are generally not guaranteed to be interpretable; that is, a decision maker may not be able to easily see the link between the current system state and the policy's action…
Learning Interpretable Point-Based Clinical Risk Scores via Direct Optimization
Department of Biomedical Data Science, Stanford University（斯坦福大学生物医学数据科学系）；Decatur High School（德凯高中）；Department of Pathology, Stanford University School of Medicine（斯坦福大学医学院病理学系）；Division of Computational Medicine, Department of Medicine, Stanford University（斯坦福大学医学系计算医学分会）
RO-FIGS: Efficient and Expressive Tree-Based Ensembles for Tabular Data
Department of Computer Science and Technology, University of Cambridge（计算机科学与技术系，剑桥大学）；U.S. Army Medical Research and Development Command（美国陆军医疗研究与开发指挥部）
University of Chicago（芝加哥大学）；Robert Bosch LLC（罗伯特·博世有限责任公司）
In this paper we introduce a novel family of decision lists consisting of highly interpretable models which can be learned efficiently in a greedy manner. The defining property is that all rules are oriented in the same direction. Particular examples of this family are decision lists with monotonically decreasing (or increasing) probabilities. On simulated data we empirically confirm that the proposed model family is easier to train than general decision lists. We exemplify the practical usability of our approach by identifying problem symptoms in a manufacturing process.
Interpretable Representation Learning for Additive Rule Ensembles
Department of Data Science and Artificial Intelligence, Monash University（墨尔本大学数据科学与人工智能系）；Department of Information Systems, University of Haifa（海法大学信息系统系）
Reinforcement learning techniques achieved human-level performance in several tasks in the last decade. However, in recent years, the need for interpretability emerged: we want to be able to understand how a system works and the reasons behind its decisions. Not only we need interpretability to assess the safety of the produced systems, we also need it to extract knowledge about unknown problems. While some techniques that optimize decision trees for reinforcement learning do exist, they usually employ greedy algorithms or they do not exploit the rewards given by the environment. This means th…
Less is More: Efficient Black-box Attribution via Minimal Interpretable Subset Selection
Institute of Information Engineering, Chinese Academy of Sciences（中国科学院信息工程研究所）；University of Chinese Academy of Sciences（中国科学院大学）；College of Computing and Data Science, Nanyang Technological University（南洋理工大学计算机与数据科学学院）；School of Artificial Intelligence, University of Science and Technology Beijing（北京科技大学人工智能学院）；Department of Mechanical Engineering, Imperial College London（伦敦帝国理工学院机械工程系）；Center for Machine Vision and Signal Analysis (CMVS), University of Oulu（奥卢大学机器视觉与信号分析中心）；School of Cyber Science and Technology, Shenzhen Campus of Sun Yat-sen University（中山大学深圳校区计算机科学与技术学院）
Insight Centre for Data Analytics（洞察数据分析中心）；University College Dublin（都柏林大学学院）
The time series classification literature has expanded rapidly over the last decade, with many new classification approaches published each year. The research focus has mostly been on improving the accuracy and efficiency of classifiers, while their interpretability has been somewhat neglected. Classifier interpretability has become a critical constraint for many application domains and the introduction of the 'right to explanation' GDPR EU legislation in May 2018 is likely to further emphasize the importance of explainable learning algorithms. In this work we analyse the state-of-the-art for …
Deep neural networks have been proven powerful at processing perceptual data, such as images and audio. However for tabular data, tree-based models are more popular. A nice property of tree-based models is their natural interpretability. In this work, we present Deep Neural Decision Trees (DNDT) -- tree models realised by neural networks. A DNDT is intrinsically interpretable, as it is a tree. Yet as it is also a neural network (NN), it can be easily implemented in NN toolkits, and trained with gradient descent rather than greedy splitting. We evaluate DNDT on several tabular datasets, verify …
One-class Classification (OCC) is an area of machine learning which addresses prediction based on unbalanced datasets. Basically, OCC algorithms achieve training by means of a single class sample, with potentially some additional counter-examples. The current OCC models give satisfaction in terms of performance, but there is an increasing need for the development of interpretable models. In the present work, we propose a one-class model which addresses concerns of both performance and interpretability. Our hybrid OCC method relies on density estimation as part of a tree-based learning algorith…
Decision trees have been widely used as classifiers in many machine learning applications thanks to their lightweight and interpretable decision process. This paper introduces Tree in Tree decision graph (TnT), a framework that extends the conventional decision tree to a more generic and powerful directed acyclic graph. TnT constructs decision graphs by recursively growing decision trees inside the internal or leaf nodes instead of greedy training. The time complexity of TnT is linear to the number of nodes in the graph, and it can construct decision graphs on large datasets. Compared to decis…
Department of Computer Science, Duke University, Durham, USA（杜克大学计算机科学系）；Department of Computer Science, University of British Columbia, Vancouver, Canada（不列颠哥伦比亚大学计算机科学系）
Department of Statistics, Texas A&M University（得克萨斯农工大学统计系）；McCombs School of Business, The University of Texas at Austin（得克萨斯大学奥斯汀分校麦克库姆斯商学院）
A decision tree is commonly restricted to use a single hyperplane to split the covariate space at each of its internal nodes. It often requires a large number of nodes to achieve high accuracy, hurting its interpretability. In this paper, we propose convex polytope trees (CPT) to expand the family of decision trees by an interpretable generalization of their decision boundary. The splitting function at each node of CPT is based on the logical disjunction of a community of differently weighted probabilistic linear decision-makers, which also geometrically corresponds to a convex polytope in the…
University of British Columbia（不列颠哥伦比亚大学）；Duke University（杜克大学）
Decision tree optimization is notoriously difficult from a computational perspective but essential for the field of interpretable machine learning. Despite efforts over the past 40 years, only recently have optimization breakthroughs been made that have allowed practical algorithms to find optimal decision trees. These new techniques have the potential to trigger a paradigm shift where it is possible to construct sparse decision trees to efficiently optimize a variety of objective functions without relying on greedy splitting and pruning heuristics that often lead to suboptimal solutions. The …
Online Learning of Decision Trees with Thompson Sampling
LIX, Ecole Polytechnique, IP Paris（巴黎综合理工学院LIX实验室，巴黎理工学院）；AI Institute, University of Waikato（怀卡托大学人工智能研究所）；LTCI, Télécom Paris, IP Paris（巴黎高等电信学院LTCI实验室，巴黎理工学院）
