Automated machine learning (AutoML) systems aim to enable training machine learning (ML) models for non-ML experts. A shortcoming of these systems is that when they fail to produce a model with high accuracy, the user has no path to improve the model other than hiring a data scientist or learning ML -- this defeats the purpose of AutoML and limits its adoption. We introduce an interpretable data feedback solution for AutoML. Our solution suggests new data points for the user to label (without requiring a pool of unlabeled data) to improve the model's accuracy. Our solution analyzes how features influence the prediction among all ML models in an AutoML ensemble, and we suggest more data samples from feature ranges that have high variance in such analysis. Our evaluation shows that our solution can improve the accuracy of AutoML by 7-8% and significantly outperforms popular active learning solutions in data efficiency, all the while providing the added benefit of being interpretable.
University of Torontoland（多伦多兰大学）；University of Edenborrow（伊登伯勒大学）
Automated Machine Learning (AutoML) is the problem of automatically finding the pipeline with the best generalization performance on some given dataset. AutoML has received enormous attention in the last decade and has been addressed with sophisticated black-box optimization techniques such as Bayesian Optimization, Grammar-Based Genetic Algorithms, and tree search algorithms. In contrast to those approaches, we present Naive AutoML, a very simple solution to AutoML that exploits important meta-knowledge about machine learning problems and makes simplifying, yet, effective assumptions to quick…
In practice, machine learning (ML) workflows require various different steps, from data preprocessing, missing value imputation, model selection, to model tuning as well as model evaluation. Many of these steps rely on human ML experts. AutoML - the field of automating these ML pipelines - tries to help practitioners to apply ML off-the-shelf without any expert knowledge. Most modern AutoML systems like auto-sklearn, H20-AutoML or TPOT aim for high predictive performance, thereby generating ensembles that consist almost exclusively of black-box models. This, in turn, makes the interpretation f…
Auto-ADMET: An Effective and Interpretable AutoML Method for Chemical ADMET Property Prediction
Baker Heart and Diabetes Institute（贝克心脏与糖尿病研究所）；School of Chemistry & Molecular Biosciences, The University of Queensland（昆士兰大学化学与分子生物科学学院）；Department of Cardiometabolic Health, The University of Melbourne（墨尔本大学心脏代谢健康系）
Predicting Mortgage Default with Machine Learning: AutoML, Class Imbalance, and Leakage Control
Amazon Web Services, Inc.（亚马逊网络服务公司）；University of Illinois Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）；University of Connecticut（康涅狄格大学）；University at Buffalo, SUNY（布法罗大学）
Leibniz University Hannover（汉诺威莱布尼茨大学）；University of Freiburg（弗赖堡大学）；Bosch Center for Artificial Intelligence（博世人工智能中心）
Automated Machine Learning (AutoML) is used more than ever before to support users in determining efficient hyperparameters, neural architectures, or even full machine learning pipelines. However, users tend to mistrust the optimization process and its results due to a lack of transparency, making manual tuning still widespread. We introduce DeepCAVE, an interactive framework to analyze and monitor state-of-the-art optimization procedures for AutoML easily and ad hoc. By aiming for full and accessible transparency, DeepCAVE builds a bridge between users and AutoML and contributes to establishi…
UniAutoML: A Human-Centered Framework for Unified Discriminative and Generative AutoML with Large Language Models
Glasgow College, University of Electronic Science and Technology of China（电子科技大学格拉斯哥学院）；School of Engineering, University of Glasgow（格拉斯哥大学工程学院）；Toursun Synbio（同舜合成生物）；Aerospace Information Research Institute, Chinese Academy of Sciences（中国科学院空天信息创新研究院）；School of Electronic, Electrical and Communication Engineering, University of Chinese Academy of Sciences（中国科学院大学电子电气与通信工程学院）；School of Information and Software Engineering, University of Electronic Science and Technology Of China（电子科技大学信息与软件工程学院）；Research Institute, Ubtech Robotics Corp（优必选机器人公司研究院）；Qiyuan Lab（启源实验室）；Department of Computer Science, Johns Hopkins University（约翰斯·霍普金斯大学计算机科学系）
Deutsches Forschungszentrum für Künstliche Intelligenz GmbH.（德国人工智能研究中心有限公司）；Institute of Informatics, Ludwig-Maximilians-Universität München（慕尼黑大学信息学院）
Automated Machine Learning in Radiomics: A Comparative Evaluation of Performance, Efficiency and Accessibility
University of Valencia（瓦伦西亚大学）；Research & Frontiers in AI Department, Quantitative Imaging Biomarkers in Medicine, Quibim SL（研究与前沿人工智能部门、定量影像生物标志物在医学中的应用、Quibim SL）；Intelligent Data Analysis Laboratory, IDAL, University of Valencia（智能数据分析实验室，IDAL，瓦伦西亚大学）
AutoML systems are currently rising in popularity, as they can build powerful models without human oversight. They often combine techniques from many different sub-fields of machine learning in order to find a model or set of models that optimize a user-supplied criterion, such as predictive performance. The ultimate goal of such systems is to reduce the amount of time spent on menial tasks, or tasks that can be solved better by algorithms while leaving decisions that require human intelligence to the end-user. In recent years, the importance of other criteria, such as fairness and interpretab…
This paper presents an experimental comparison among four Automated Machine Learning (AutoML) methods for recommending the best classification algorithm for a given input dataset. Three of these methods are based on Evolutionary Algorithms (EAs), and the other is Auto-WEKA, a well-known AutoML method based on the Combined Algorithm Selection and Hyper-parameter optimisation (CASH) approach. The EA-based methods build classification algorithms from a single machine learning paradigm: either decision-tree induction, rule induction, or Bayesian network classification. Auto-WEKA combines algorithm…
While the demand for machine learning (ML) applications is booming, there is a scarcity of data scientists capable of building such models. Automatic machine learning (AutoML) approaches have been proposed that help with this problem by synthesizing end-to-end ML data processing pipelines. However, these follow a best-effort approach and a user in the loop is necessary to curate and refine the derived pipelines. Since domain experts often have little or no expertise in machine learning, easy-to-use interactive interfaces that guide them throughout the model building process are necessary. In t…
SELA: Tree-Search Enhanced LLM Agents for Automated Machine Learning
DeepWisdom（深度智慧）；University of California, Berkeley（加州大学伯克利分校）；The Hong Kong University of Science and Technology (Guangzhou)（香港科技大学（广州））；University of California, San Diego（加州大学圣迭戈分校）；South China Normal University（华南师范大学）；Stanford University（斯坦福大学）；The Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳））；Université de Montréal（蒙特利尔大学）；Mila（米拉研究所）
University of Georgia（佐治亚大学）；Kennesaw State University（肯尼索州立大学）
Statistical methods such as the Box-Jenkins method for time-series forecasting have been prominent since their development in 1970. Many researchers rely on such models as they can be efficiently estimated and also provide interpretability. However, advances in machine learning research indicate that neural networks can be powerful data modeling techniques, as they can give higher accuracy for a plethora of learning problems and datasets. In the past, they have been tried on time-series forecasting as well, but their overall results have not been significantly better than the statistical model…
Cedars Sinai Medical Center（西达赛奈医学中心）；University of Pennsylvania（宾夕法尼亚大学）
Machine learning (ML) offers powerful methods for detecting and modeling associations often in data with large feature spaces and complex associations. Many useful tools/packages (e.g. scikit-learn) have been developed to make the various elements of data handling, processing, modeling, and interpretation accessible. However, it is not trivial for most investigators to assemble these elements into a rigorous, replicatable, unbiased, and effective data analysis pipeline. Automated machine learning (AutoML) seeks to address these issues by simplifying the process of ML analysis for all. Here, we…
In recent years, a large number of XAI (eXplainable Artificial Intelligence) solutions have been proposed to explain existing ML (Machine Learning) models or to create interpretable ML models. Evaluation measures have recently been proposed and it is now possible to compare these XAI solutions. However, selecting the most relevant XAI solution among all this diversity is still a tedious task, especially when meeting specific needs and constraints. In this paper, we propose AutoXAI, a framework that recommends the best XAI solution and its hyperparameters according to specific XAI evaluation me…
John A. Paulson School of Engineering and Applied Sciences（约翰·A·保尔森工程与应用科学学院）；Harvard University（哈佛大学）
Machine learning models that first learn a representation of a domain in terms of human-understandable concepts, then use it to make predictions, have been proposed to facilitate interpretation and interaction with models trained on high-dimensional data. However these methods have important limitations: the way they define concepts are not inherently interpretable, and they assume that concept labels either exist for individual instances or can easily be acquired from users. These limitations are particularly acute for high-dimensional tabular features. We propose an approach for learning a s…
University of Bristol（布里斯托大学）；CortAIx, Thales（泰雷兹公司 CortAIx）
The potential of reinforcement learning (RL) to deliver aligned and performant agents is partially bottlenecked by the reward engineering problem. One alternative to heuristic trial-and-error is preference-based RL (PbRL), where a reward function is inferred from sparse human feedback. However, prior PbRL methods lack interpretability of the learned reward structure, which hampers the ability to assess robustness and alignment. We propose an online, active preference learning algorithm that constructs reward functions with the intrinsically interpretable, compositional structure of a tree. Usi…
Closing the Loop: PID Feedback Control for Interpretable Activation Steering in Symbolic Music Generation
Athens University of Economics and Business（雅典经济与商业大学）；Orfium Research（Orfium 研究）；Hellenic Mediterranean University（希腊地中海大学）；Archimedes / Athena Research Center（阿基米德/雅典娜研究中心）
Stevens Institute of Technology（斯蒂文斯理工学院）；Sapienza University of Rome（罗马第一大学）；Institute of Applied Computing, CNR（意大利国家研究委员会应用计算研究所）
Probabilistic password strength meters have been proved to be the most accurate tools to measure password strength. Unfortunately, by construction, they are limited to solely produce an opaque security estimation that fails to fully support the user during the password composition. In the present work, we move the first steps towards cracking the intelligibility barrier of this compelling class of meters. We show that probabilistic password meters inherently own the capability of describing the latent relation occurring between password strength and password structure. In our approach, the sec…
Kasetsart University（农业大学）；Mahidol University（玛希隆大学）；Vidyasirimedhi Institute of Science & Technology（维迪亚西里梅迪科学技术研究所）
In the past few years, there are several researches on Parkinson's disease (PD) recognition using single-photon emission computed tomography (SPECT) images with deep learning (DL) approach. However, the DL model's complexity usually results in difficult model interpretation when used in clinical. Even though there are multiple interpretation methods available for the DL model, there is no evidence of which method is suitable for PD recognition application. This tutorial aims to demonstrate the procedure to choose a suitable interpretation method for the PD recognition model. We exhibit four DC…
