Random forests are considered one of the best out-of-the-box classification and regression algorithms due to their high level of predictive performance with relatively little tuning. Pairwise proximities can be computed from a trained random forest and measure the similarity between data points relative to the supervised task. Random forest proximities have been used in many applications including the identification of variable importance, data imputation, outlier detection, and data visualization. However, existing definitions of random forest proximities do not accurately reflect the data geometry learned by the random forest. In this paper, we introduce a novel definition of random forest proximities called Random Forest-Geometry- and Accuracy-Preserving proximities (RF-GAP). We prove that the proximity-weighted sum (regression) or majority vote (classification) using RF-GAP exactly matches the out-of-bag random forest prediction, thus capturing the data geometry learned by the random forest. We empirically show that this improved geometric representation outperforms traditional random forest proximities in tasks such as data imputation and provides outlier detection and visualization results consistent with the learned data geometry.
In the present paper we present a surrogate model, which can be used to estimate extreme tower loads on a wind turbine from a number of signals and a suitable simulation tool. Due to the requirements of the International Electrotechnical Commission (IEC) Standard 61400-1, assessing extreme tower loads on wind turbines constitutes a key component of the design phase. The proposed model imputes tower loads by matching observed signals with simulated quantities using proximities induced by random forests. In this way the algorithm's adaptability to high-dimensional and sparse settings is exploite…
Dept. of Mathematics \& Statistics Utah State University Logan, Utah, USA；Dept. of Computer Science Brigham Young University Provo, UT, USA；Dept. of Statistics Brigham Young University Provo, UT, USA；Dept. of Mathematics \& Statistics Utah State University Logan, UT, USA
ISI Foundation（ISI基金会）；University of Wisconsin-Madison（威斯康星大学麦迪逊分校）
We study the problem of privacy-preserving machine learning (PPML) for ensemble methods, focusing our effort on random forests. In collaborative analysis, PPML attempts to solve the conflict between the need for data sharing and privacy. This is especially important in privacy sensitive applications such as learning predictive models for clinical decision support from EHR data from different clinics, where each clinic has a responsibility for its patients' privacy. We propose a new approach for ensemble methods: each entity learns a model, from its own data, and then when a client asks the pre…
Label-Guided Imputation via Forest-Based Proximities for Improved Time Series Classification
1 Department of Statistics, Brigham Young University, Provo, UT, USA Emails；2 Department of Computer Science, Brigham Young University, Provo, UT, USA Email；3 Department of Public Health, Brigham Young University, Provo, UT, USA Email；4 Department of Neurology, Vagelos College of Physicians \& Surgeons, Columbia University, New York, NY, USA Email；5 Department of Neurology, Miller School of Medicine, University of Miami, Miami, FL, USA Email；6 Elmhurst Hospital Center, NYC Health + Hospitals, New York, NY, USA Email；7 Department of Mathematics \& Statistics, Utah State University, Logan, UT, USA Email
JD Intelligent Cities Research（京东智能城市研究院）；JD Intelligent Cities Business Unit（京东智能城市事业部）；University of Science and Technology of China（中国科学技术大学）；Beijing Normal University（北京师范大学）
Most real-world data are scattered across different companies or government organizations, and cannot be easily integrated under data privacy and related regulations such as the European Union's General Data Protection Regulation (GDPR) and China' Cyber Security Law. Such data islands situation and data privacy & security are two major challenges for applications of artificial intelligence. In this paper, we tackle these challenges and propose a privacy-preserving machine learning model, called Federated Forest, which is a lossless learning model of the traditional random forest method, i.e., …
Utah State University（犹他州立大学）；Université de Montréal（蒙特利尔大学）
Dimensionality reduction is often used as an initial step in data exploration, either as preprocessing for classification or regression or for visualization. Most dimensionality reduction techniques to date are unsupervised; they do not take class labels into account (e.g., PCA, MDS, t-SNE, Isomap). Such methods require large amounts of data and are often sensitive to noise that may obfuscate important patterns in the data. Various attempts at supervised dimensionality reduction methods that take into account auxiliary annotations (e.g., class labels) have been successfully implemented with go…
We present two new statistical machine learning methods designed to learn on fully homomorphic encrypted (FHE) data. The introduction of FHE schemes following Gentry (2009) opens up the prospect of privacy preserving statistical machine learning analysis and modelling of encrypted data without compromising security constraints. We propose tailored algorithms for applying extremely random forests, involving a new cryptographic stochastic fraction estimator, and naïve Bayes, involving a semi-parametric model for the class decision boundary, and show how they can be used to learn and predict from…
Random Forests (RF) is a popular machine learning method for classification and regression problems. It involves a bagging application to decision tree models. One of the primary advantages of the Random Forests model is the reduction in the variance of the forecast. In large scale applications of the model with millions of data points and hundreds of features, the size of the fitted objects can get very large and reach the limits on the available space in production setups, depending on the number and depth of the trees. This could be especially challenging when trained models need to be down…
University of Applied Sciences Würzburg-Schweinfurt（维尔茨堡-施韦因富特应用科学大学）；University of Groningen（格罗宁根大学）
Proximities are at the heart of almost all machine learning methods. If the input data are given as numerical vectors of equal lengths, euclidean distance, or a Hilbertian inner product is frequently used in modeling algorithms. In a more generic view, objects are compared by a (symmetric) similarity or dissimilarity measure, which may not obey particular mathematical properties. This renders many machine learning methods invalid, leading to convergence problems and the loss of guarantees, like generalization bounds. In many cases, the preferred dissimilarity measure is not metric, like the ea…
Can synthetic data reproduce real-world findings in epidemiology? A replication study using adversarial random forests
Leibniz Institute for Prevention Research and Epidemiology—BIPS（预防研究与流行病学研究所—BIPS）；Faculty of Mathematics and Computer Science, University of Bremen（不莱梅大学数学与计算机科学学院）；Department of Human Health and Nutritional Sciences, University of Guelph（圭尔夫大学人类健康与营养科学系）；Institute of Epidemiology and Social Medicine, University of Münster（明斯特大学流行病学与社会医学研究所）；Institute of General Practice/Family Medicine, Faculty of Medicine and Medical Center, University of Freiburg（弗赖堡大学内科/家庭医学研究所）；Freiburg Center for Data Analysis, Modeling and AI, University of Freiburg（弗赖堡数据分析、建模与人工智能中心）；German Cancer Research Center (DKFZ), Heidelberg, Germany（德国癌症研究中心（DKFZ））；Department of Epidemiology, Helmholtz Centre for Infection Research (HZI), Brunswick, Germany（感染研究哈勒研究所流行病学部）；Department of Epidemiology and Preventive Medicine, University of Regensburg（雷根施塔特大学流行病学与预防医学系）；Institute for Occupational and Maritime Medicine (ZfAM), University Medical Center Hamburg-Eppendorf（汉堡-埃彭多夫大学医学中心职业与航海医学研究所）；Saarland Cancer Registry, Saarbrücken, Germany（萨尔州癌症登记处）
Applying machine learning algorithms to private data, such as financial or medical data, while preserving their confidentiality, is a difficult task. Homomorphic Encryption (HE) is acknowledged for its ability to allow computation on encrypted data, where both the input and output are encrypted, which therefore enables secure inference on private data. Nonetheless, because of the constraints of HE, such as its inability to evaluate non-polynomial functions or to perform arbitrary matrix multiplication efficiently, only inference of linear models seem usable in practice in the HE paradigm so fa…
The University of Hong Kong（香港大学）；Wells Fargo（富国银行）
Prediction accuracy and model explainability are the two most important objectives when developing machine learning algorithms to solve real-world problems. The neural networks are known to possess good prediction performance, but lack of sufficient model interpretability. In this paper, we propose to enhance the explainability of neural networks through the following architecture constraints: a) sparse additive subnetworks; b) projection pursuit with orthogonality constraint; and c) smooth function approximation. It leads to an explainable neural network (xNN) with the superior balance betwee…
In this paper we consider the problem of graph-based transductive classification, and we are particularly interested in the directed graph scenario which is a natural form for many real world applications. Different from existing research efforts that either only deal with undirected graphs or circumvent directionality by means of symmetrization, we propose a novel random walk approach on directed graphs using absorbing Markov chains, which can be regarded as maximizing the accumulated expected number of visits from the unlabeled transient states. Our algorithm is simple, easy to implement, an…
In oncology clinical trials, characterizing the long-term overall survival (OS) benefit for an experimental drug or treatment regimen (experimental group) is often unobservable if some patients in the control group switch to drugs in the experimental group and/or other cancer treatments after disease progression. A key question often raised by payers and reimbursement agencies is how to estimate the true benefit of the experimental drug group on overall survival that would have been estimated if there were no treatment switches. Several commonly used statistical methods are available to estima…
To further develop the statistical inference problem for heterogeneous treatment effects, this paper builds on Breiman's (2001) random forest tree (RFT)and Wager et al.'s (2018) causal tree to parameterize the nonparametric problem using the excellent statistical properties of classical OLS and the division of local linear intervals based on covariate quantile points, while preserving the random forest trees with the advantages of constructible confidence intervals and asymptotic normality properties [Athey and Imbens (2016),Efron (2014),Wager et al.(2014)\citep{wager2014asymptotic}], we propo…
Estimation of forest height and biomass from open-access multi-sensor satellite imagery and GEDI Lidar data: high-resolution maps of metropolitan France
