Finding a well-performing architecture is often tedious for both DL practitioners and researchers, leading to tremendous interest in the automation of this task by means of neural architecture search (NAS). Although the community has made major strides in developing better NAS methods, the quality of scientific empirical evaluations in the young field of NAS is still lacking behind that of other areas of machine learning. To address this issue, we describe a set of possible issues and ways to avoid them, leading to the NAS best practices checklist available at http://automl.org/nas_checklist.pdf.
Project Jupyter（Jupyter项目）；Facebook AI Research（脸书人工智能研究院）
In the quest to align deep learning with the sciences to address calls for rigor, safety, and interpretability in machine learning systems, this contribution identifies key missing pieces: the stages of hypothesis formulation and testing, as well as statistical and systematic uncertainty estimation -- core tenets of the scientific method. This position paper discusses the ways in which contemporary science is conducted in other domains and identifies potentially useful practices. We present a case study from physics and describe how this field has promoted rigor through specific methodological…
Best Practices for Machine Learning Experimentation in Scientific Applications
Computer Science Department, Lucerne University of Applied Sciences and Arts（卢塞恩应用科学与艺术大学计算机科学系）；Institute of Applied Mathematics and Physics（应用数学与物理研究所）；ZHAW - Zurich University of Applied Sciences（苏黎世应用科学大学）
Unsupervised Machine Learning for Scientific Discovery: Workflow and Best Practices
Department of Neuroscience, Baylor College of Medicine（神经科学系，贝勒医学院）；Department of Applied and Computational Mathematics and Statistics, University of Notre Dame（应用与计算数学和统计学系，圣母大学）；Department of Statistics, Columbia University（统计学系，哥伦比亚大学）；Zuckerman Institute & Irving Institute, Columbia University（Zuckerman研究所与Irving研究所，哥伦比亚大学）
Verification and Validation for Trustworthy Scientific Machine Learning
Sandia National Laboratories（桑迪亚国家实验室）；The George Washington University（乔治·华盛顿大学）；University of Michigan（密歇根大学）；The University of Texas at Austin（德克萨斯大学奥斯汀分校）
Despite the widespread adoption of computational notebooks, little is known about best practices for their usage in collaborative contexts. In this paper, we fill this gap by eliciting a catalog of best practices for collaborative data science with computational notebooks. With this aim, we first look for best practices through a multivocal literature review. Then, we conduct interviews with professional data scientists to assess their awareness of these best practices. Finally, we assess the adoption of best practices through the analysis of 1,380 Jupyter notebooks retrieved from the Kaggle p…
Fraunhofer Heinrich Hertz Institute（弗劳恩霍夫海因里希赫兹研究所）；Technische Universität Berlin（柏林工业大学）；Singapore University of Technology and Design（新加坡科技设计大学）
Within the last decade, neural network based predictors have demonstrated impressive - and at times super-human - capabilities. This performance is often paid for with an intransparent prediction process and thus has sparked numerous contributions in the novel field of explainable artificial intelligence (XAI). In this paper, we focus on a popular and widely used method of XAI, the Layer-wise Relevance Propagation (LRP). Since its initial proposition LRP has evolved as a method, and a best practice for applying the method has tacitly emerged, based however on humanly observed evidence alone. I…
Universidad de los Andes（洛斯安第斯大学）；Technische Universität Clausthal（克劳斯塔尔工业大学）；Universidad Nacional（国立大学）
Machine Learning (ML) is being used in multiple disciplines due to its powerful capability to infer relationships within data. In particular, Software Engineering (SE) is one of those disciplines in which ML has been used for multiple tasks, like software categorization, bugs prediction, and testing. In addition to the multiple ML applications, some studies have been conducted to detect and understand possible pitfalls and issues when using ML. However, to the best of our knowledge, only a few studies have focused on presenting ML best practices or guidelines for the application of ML in diffe…
Karlsruhe Institute of Technology (KIT)（卡尔斯鲁厄理工学院（KIT））
Given a set of empirical observations, conditional density estimation aims to capture the statistical relationship between a conditional variable $\mathbf{x}$ and a dependent variable $\mathbf{y}$ by modeling their conditional probability $p(\mathbf{y}|\mathbf{x})$. The paper develops best practices for conditional density estimation for finance applications with neural networks, grounded on mathematical insights and empirical evaluations. In particular, we introduce a noise regularization and data normalization scheme, alleviating problems with over-fitting, initialization and hyper-parameter…
University at Buffalo, The State University of New York（纽约州立大学布法罗分校）；New York State Center of Excellence in Materials Informatics（纽约州材料信息学卓越中心）
This review aims to draw attention to two issues of concern when we set out to make machine learning work in the chemical and materials domain, i.e., statistical loss function metrics for the validation and benchmarking of data-derived models, and the uncertainty quantification of predictions made by them. They are often overlooked or underappreciated topics as chemists typically only have limited training in statistics. Aside from helping to assess the quality, reliability, and applicability of a given model, these metrics are also key to comparing the performance of different models and thus…
Biological neural networks continue to inspire breakthroughs in neural network performance. And yet, one key area of neural computation that has been under-appreciated and under-investigated is biologically plausible, energy-efficient spiking neural networks, whose potential is especially attractive for low-power, mobile, or otherwise hardware-constrained settings. We present a literature review of recent developments in the interpretation, optimization, efficiency, and accuracy of spiking neural networks. Key contributions include identification, discussion, and comparison of cutting-edge met…
On the Readiness of Scientific Data for a Fair and Transparent Use in Machine Learning
Internet Interdisciplinary Institute (IN3)（互联网跨学科研究所（IN3））；Universitat Oberta de Catalunya (UOC)（加泰罗尼亚开放大学（UOC））；Barcelona Supercomputing Center（巴塞罗那超级计算中心）；Luxembourg Institute of Science and Technology（卢森堡科学技术研究院）；University of Luxembourg（卢森堡大学）
Delft University of Technology（代尔夫特理工大学）；Energyville（能源城（能源研究机构））；Flemish Institute for Technological Research (VITO)（弗拉芒技术研究院）；Wrocław University of Science and Technology（弗罗茨瓦夫理工大学）
While the field of electricity price forecasting has benefited from plenty of contributions in the last two decades, it arguably lacks a rigorous approach to evaluating new predictive algorithms. The latter are often compared using unique, not publicly available datasets and across too short and limited to one market test samples. The proposed new methods are rarely benchmarked against well established and well performing simpler models, the accuracy metrics are sometimes inadequate and testing the significance of differences in predictive performance is seldom conducted. Consequently, it is n…
Identifying Best Practice Melting Patterns in Induction Furnaces: A Data-Driven Approach Using Time Series KMeans Clustering and Multi-Criteria Decision Making
