Automated machine learning has gained a lot of attention recently. Building and selecting the right machine learning models is often a multi-objective optimization problem. General purpose machine learning software that simultaneously supports multiple objectives and constraints is scant, though the potential benefits are great. In this work, we present a framework called Autotune that effectively handles multiple objectives and constraints that arise in machine learning problems. Autotune is built on a suite of derivative-free optimization methods, and utilizes multi-level parallelism in a distributed computing environment for automatically training, scoring, and selecting good models. Incorporation of multiple objectives and constraints in the model exploration and selection process provides the flexibility needed to satisfy trade-offs necessary in practical machine learning applications. Experimental results from standard multi-objective optimization benchmark problems show that Autotune is very efficient in capturing Pareto fronts. These benchmark results also show how adding constraints can guide the search to more promising regions of the solution space, ultimately producing more desirable Pareto fronts. Results from two real-world case studies demonstrate the effectiveness of the constrained multi-objective optimization capability offered by Autotune.
Laboratory of Innovations in Transportation (LiTrans)（交通创新实验室）；Ryerson University（瑞尔森大学）
This study exploits the advancements in information and communication technology (ICT), connected and automated vehicles (CAVs), and sensing, to develop proactive multi-objective eco-routing strategies. For a robust application, several GHG costing approaches are examined. The predictive models for the link level traffic and emission states are developed using long short term memory deep network with exogenous predictors. It is found that proactive routing strategies outperformed the myopic strategies, regardless of the routing objective. Whether myopic or proactive, the multi-objective routin…
REsearch Groups in Intelligent Machines（智能机器研究组）；University of Sfax（斯法克斯大学）；ENIS（斯法克斯国立工程师学院）
Feature Selection (FS) plays an important role in learning and classification tasks. The object of FS is to select the relevant and non-redundant features. Considering the huge amount number of features in real-world applications, FS methods using batch learning technique can't resolve big data problem especially when data arrive sequentially. In this paper, we propose an online feature selection system which resolves this problem. More specifically, we treat the problem of online supervised feature selection for binary classification as a decision-making problem. A philosophical vision to thi…
This paper addresses the problem of constrained multi-objective optimization over black-box objective functions with practitioner-specified preferences over the objectives when a large fraction of the input space is infeasible (i.e., violates constraints). This problem arises in many engineering design problems including analog circuits and electric power system design. Our overall goal is to approximate the optimal Pareto set over the small fraction of feasible input designs. The key challenges include the huge size of the design space, multiple objectives and large number of constraints, and…
Not All Objectives Are Born Equal: Priority-Constrained Descent for Hierarchical Multi-Objective Optimization
Department of Computer Science & Engineering, American University of Sharjah（美国沙迦大学计算机科学与工程系）；Senseable City Laboratory, Massachusetts Institute of Technology（麻省理工学院可感知城市实验室）
Sandia National Laboratories（桑迪亚国家实验室）；Xilinx Inc.（赛灵思公司）；Georgia Institute of Technology（佐治亚理工学院）
Bayesian optimization (BO) is an efficient and flexible global optimization framework that is applicable to a very wide range of engineering applications. To leverage the capability of the classical BO, many extensions, including multi-objective, multi-fidelity, parallelization, latent-variable model, have been proposed to improve the limitation of the classical BO framework. In this work, we propose a novel multi-objective (MO) extension, called srMO-BO-3GP, to solve the MO optimization problems in a sequential setting. Three different Gaussian processes (GPs) are stacked together, where each…
Xi’an Jiaotong University（西安交通大学）；University of Kansas Medical Center（堪萨斯大学医学中心）
Accurately identifying patient's status through medical images plays an important role in diagnosis and treatment. Artificial intelligence (AI), especially the deep learning, has achieved great success in many fields. However, more reliable AI model is needed in image guided diagnosis and therapy. To achieve this goal, developing a balanced, safe and robust model with a unified framework is desirable. In this study, a new unified model termed as automated multi-objective Mixer (AutoMO-Mixer) model was developed, which utilized a recent developed multiple layer perceptron Mixer (MLP-Mixer) as b…
In this paper, we introduce an Augmented Lagrangian based method to incorporate the multiple objectives (MO) in a search ranking algorithm. Optimizing MOs is an essential and realistic requirement for building ranking models in production. The proposed method formulates MO in constrained optimization and solves the problem in the popular Boosting framework -- a novel contribution of our work. Furthermore, we propose a procedure to set up all optimization parameters in the problem. The experimental results show that the method successfully achieves MO criteria much more efficiently than existin…
We consider a popular family of constrained optimization problems arising in machine learning that involve optimizing a non-decomposable evaluation metric with a certain thresholded form, while constraining another metric of interest. Examples of such problems include optimizing the false negative rate at a fixed false positive rate, optimizing precision at a fixed recall, optimizing the area under the precision-recall or ROC curves, etc. Our key idea is to formulate a rate-constrained optimization that expresses the threshold parameter as a function of the model parameters via the Implicit Fu…
Learn Feasibility Once, Optimize All Objectives: Derivative-Free Diffusion Models for Chance-Constrained Programming
Beijing University of Posts and Telecommunications（北京邮电大学）；Nankai University（南开大学）；University of the Chinese Academy of Sciences（中国科学院大学）；Southwest University of Finance and Economics（西南财经大学）
Imperial College London（帝国理工学院）；BASF SE（巴斯夫公司）；Cooperative State University Mannheim（曼海姆合作州立大学）；University College London（伦敦大学学院）
Energy systems optimization problems are complex due to strongly non-linear system behavior and multiple competing objectives, e.g. economic gain vs. environmental impact. Moreover, a large number of input variables and different variable types, e.g. continuous and categorical, are challenges commonly present in real-world applications. In some cases, proposed optimal solutions need to obey explicit input constraints related to physical properties or safety-critical operating conditions. This paper proposes a novel data-driven strategy using tree ensembles for constrained multi-objective optim…
École Polytechnique（巴黎综合理工学院）；CNRS（法国国家科学研究中心）；Université Paris Diderot（巴黎第七大学）；Université Paris-Dauphine PSL（巴黎第九大学（巴黎文理研究大学））
The minimization of convex objectives coming from linear supervised learning problems, such as penalized generalized linear models, can be formulated as finite sums of convex functions. For such problems, a large set of stochastic first-order solvers based on the idea of variance reduction are available and combine both computational efficiency and sound theoretical guarantees (linear convergence rates). Such rates are obtained under both gradient-Lipschitz and strong convexity assumptions. Motivated by learning problems that do not meet the gradient-Lipschitz assumption, such as linear Poisso…
Joint Optimization of Training and Inference in Federated Edge Learning via Constrained Multi-Objective Deep Reinforcement Learning
Department of Electrical and Computer Engineering, Concordia University（康科迪亚大学电气与计算机工程系）；School of Automation, Guangdong University of Technology（广东工业大学自动化学院）
Computer Science & Artificial Intelligence Lab（计算机科学与人工智能实验室）；Massachusetts Institute of Technology（麻省理工学院）
We present AutoOED, an Optimal Experiment Design platform powered with automated machine learning to accelerate the discovery of optimal solutions. The platform solves multi-objective optimization problems in time- and data-efficient manner by automatically guiding the design of experiments to be evaluated. To automate the optimization process, we implement several multi-objective Bayesian optimization algorithms with state-of-the-art performance. AutoOED is open-source and written in Python. The codebase is modular, facilitating extensions and tailoring the code, serving as a testbed for mach…
University "Politehnica" of Bucharest（布加勒斯特理工大学）；Infineon Technologies（英飞凌科技）
With the ever increasing complexity of specifications, manual sizing for analog circuits recently became very challenging. Especially for innovative, large-scale circuits designs, with tens of design variables, operating conditions and conflicting objectives to be optimized, design engineers spend many weeks, running time-consuming simulations, in their attempt at finding the right configuration. Recent years brought machine learning and optimization techniques to the field of analog circuits design, with evolutionary algorithms and Bayesian models showing good results for circuit sizing. In t…
Domain Switching on the Pareto Front: Multi-Objective Deep Kernel Learning in Automated Piezoresponse Force Microscopy
University of Tennessee（田纳西大学）；Institute of Science Tokyo（东京科学大学）；University of California, Berkeley（加州大学伯克利分校）；Rice University（莱斯大学）；Pacific Northwest National Laboratory（太平洋西北国家实验室）
Towards Objective Gastrointestinal Auscultation: Automated Segmentation and Annotation of Bowel Sound Patterns
Division AI4Health, Department for Health Services Research, Faculty of Medicine and Health Sciences, Carl von Ossietzky Universität Oldenburg（AI4Health部门、健康服务研究部门、医学与健康科学学院、卡尔·冯·奥西特茨基大学Oldenburg分校）；Fraunhofer IDMT, Institute Part HSA（弗劳恩霍夫IDMT研究所HSA部分）；University Clinic for Visceral Surgery, Faculty of Medicine and Health Sciences, Carl von Ossietzky Universität Oldenburg（visceral surgery大学诊所、医学与健康科学学院、卡尔·冯·奥西特茨基大学Oldenburg分校）
In this study, we introduce a novel platform Resource-Aware AutoML (RA-AutoML) which enables flexible and generalized algorithms to build machine learning models subjected to multiple objectives, as well as resource and hard-ware constraints. RA-AutoML intelligently conducts Hyper-Parameter Search(HPS) as well as Neural Architecture Search (NAS) to build models optimizing predefined objectives. RA-AutoML is a versatile framework that allows user to prescribe many resource/hardware constraints along with objectives demanded by the problem at hand or business requirements. At its core, RA-AutoML…
Significant effort has been made to solve computationally expensive optimization problems in the past two decades, and various optimization methods incorporating surrogates into optimization have been proposed. Most research focuses on either exploiting the surrogate by defining a utility optimization problem or customizing an existing optimization method to use one or multiple approximation models. However, only a little attention has been paid to generic concepts applicable to different types of algorithms and optimization problems simultaneously. Thus this paper proposes a generalized proba…
Mitsubishi Electric Research Laboratories (MERL)（三菱电机研究实验室）
Safety and robustness are two desired properties for any reinforcement learning algorithm. CMDPs can handle additional safety constraints and RMDPs can perform well under model uncertainties. In this paper, we propose to unite these two frameworks resulting in robust constrained MDPs (RCMDPs). The motivation is to develop a framework that can satisfy safety constraints while also simultaneously offer robustness to model uncertainties. We develop the RCMDP objective, derive gradient update formula to optimize this objective and then propose policy gradient based algorithms. We also independentl…
