The current practice of manually processing features for high-dimensional and heterogeneous aviation data is labor-intensive, does not scale well to new problems, and is prone to information loss, affecting the effectiveness and maintainability of machine learning (ML) procedures. This research explored an unsupervised learning method, autoencoder, to extract effective features for aviation machine learning problems. The study explored variants of autoencoders with the aim of forcing the learned representations of the input to assume useful properties. A flight track anomaly detection autoencoder was developed to demonstrate the versatility of the technique. The research results show that the autoencoder can not only automatically extract effective features for the flight track data, but also efficiently deep clean data, thereby reducing the workload of data scientists. Moreover, the research leveraged transfer learning to efficiently train models for multiple airports. Transfer learning can reduce model training times from days to hours, as well as improving model performance. The developed applications and techniques are shared with the whole aviation community to improve effectiveness of ongoing and future machine learning studies.
This research focuses on predicting the demand for air taxi urban air mobility (UAM) services during different times of the day in various geographic regions of New York City using machine learning algorithms (MLAs). Several ride-related factors (such as month of the year, day of the week and time of the day) and weather-related variables (such as temperature, weather conditions and visibility) are used as predictors for four popular MLAs, namely, logistic regression, artificial neural networks, random forests, and gradient boosting. Experimental results suggest gradient boosting to consistent…
Unsupervised learning with functional data is an emerging paradigm of machine learning research with applications to computer vision, climate modeling and physical systems. A natural way of modeling functional data is by learning operators between infinite dimensional spaces, leading to discretization invariant representations that scale independently of the sample grid resolution. Here we present Variational Autoencoding Neural Operators (VANO), a general strategy for making a large class of operator learning architectures act as variational autoencoders. For this purpose, we provide a novel …
Freie Universität Berlin（柏林自由大学）；Bernstein Center for Computational Neuroscience（伯恩斯坦计算神经科学中心）；Berlin Institute of Technology（柏林工业大学）
Much work has been done refining and characterizing the receptive fields learned by deep learning algorithms. A lot of this work has focused on the development of Gabor-like filters learned when enforcing sparsity constraints on a natural image dataset. Little work however has investigated how these filters might expand to the temporal domain, namely through training on natural movies. Here we investigate exactly this problem in established temporal deep learning algorithms as well as a new learning paradigm suggested here, the Temporal Autoencoding Restricted Boltzmann Machine (TARBM).
The problem of learning a manifold structure on a dataset is framed in terms of a generative model, to which we use ideas behind autoencoders (namely adversarial/Wasserstein autoencoders) to fit deep neural networks. From a machine learning perspective, the resulting structure, an atlas of a manifold, may be viewed as a combination of dimensionality reduction and "fuzzy" clustering.
We present effective pre-training strategies for neural machine translation (NMT) using parallel corpora involving a pivot language, i.e., source-pivot and pivot-target, leading to a significant improvement in source-target translation. We propose three methods to increase the relation among source, pivot, and target languages in the pre-training: 1) step-wise training of a single model for different language pairs, 2) additional adapter component to smoothly connect pre-trained encoder and decoder, and 3) cross-lingual encoder training via autoencoding of the pivot language. Our methods great…
Commercial aviation is one of the biggest contributors towards climate change. We propose to reduce environmental impact of aviation by considering solutions that would reduce the flight time. Specifically, we first consider improving winds aloft forecast so that flight planners could use better information to find routes that are efficient. Secondly, we propose an aircraft routing method that seeks to find the fastest route to the destination by considering uncertainty in the wind forecasts and then optimally trading-off between exploration and exploitation.
University of Warwick（华威大学）；The Alan Turing Institute（艾伦·图灵研究所）
Specifying a governing physical model in the presence of missing physics and recovering its parameters are two intertwined and fundamental problems in science. Modern machine learning allows one to circumvent these, via emulators and surrogates, but in doing so disregards prior knowledge and physical laws that are especially important for small data regimes, interpretability, and decision making. In this work we fold the mechanistic model into a flexible data-driven surrogate to arrive at a physically structured decoder network. This provides accelerated inference for the Bayesian inverse prob…
Freie Universität Berlin（柏林自由大学）；Technische Universität Berlin（柏林工业大学）
Restricted Boltzmann Machines (RBMs) are generative models which can learn useful representations from samples of a dataset in an unsupervised fashion. They have been widely employed as an unsupervised pre-training method in machine learning. RBMs have been modified to model time series in two main ways: The Temporal RBM stacks a number of RBMs laterally and introduces temporal dependencies between the hidden layer units; The Conditional RBM, on the other hand, considers past samples of the dataset as a conditional bias and learns a representation which takes these into account. Here we propos…
Georgia Institute of Technology（佐治亚理工学院）；KBR Inc.（KBR公司）
In recent years, there has been a rapid growth in the application of machine learning techniques that leverage aviation data collected from commercial airline operations to improve safety. Anomaly detection and predictive maintenance have been the main targets for machine learning applications. However, this paper focuses on the identification of precursors, which is a relatively newer application. Precursors are events correlated with adverse events that happen prior to the adverse event itself. Therefore, precursor mining provides many benefits including understanding the reasons behind a sa…
We propose a novel model architecture and training algorithm to learn bilingual sentence embeddings from a combination of parallel and monolingual data. Our method connects autoencoding and neural machine translation to force the source and target sentence embeddings to share the same space without the help of a pivot language or an additional transformation. We train a multilayer perceptron on top of the sentence embeddings to extract good bilingual sentence pairs from nonparallel or noisy parallel data. Our approach shows promising performance on sentence alignment recovery and the WMT 2018 …
The Cooper Union for the Advancement of Science and Art（库珀高等科学艺术联盟学院）
A method for musical audio synthesis using autoencoding neural networks is proposed. The autoencoder is trained to compress and reconstruct magnitude short-time Fourier transform frames. The autoencoder produces a spectrogram by activating its smallest hidden layer, and a phase response is calculated using real-time phase gradient heap integration. Taking an inverse short-time Fourier transform produces the audio signal. Our algorithm is light-weight when compared to current state-of-the-art audio-producing machine learning algorithms. We outline our design process, produce metrics, and detail…
Technical University of Munich（慕尼黑工业大学）；KBR（KBR公司）；NASA Ames Research Center（美国国家航空航天局艾姆斯研究中心）
The exceptional progress in the field of machine learning (ML) in recent years has attracted a lot of interest in using this technology in aviation. Possible airborne applications of ML include safety-critical functions, which must be developed in compliance with rigorous certification standards of the aviation industry. Current certification standards for the aviation industry were developed prior to the ML renaissance without taking specifics of ML technology into account. There are some fundamental incompatibilities between traditional design assurance approaches and certain aspects of ML-b…
"Task-relevant autoencoding" enhances machine learning for human neuroscience
University of California, Irvine（加州大学欧文分校）；Université de Montréal（蒙特利尔大学）；Centre de recherche de l’institut universitaire en santé mentale de Montréal（蒙特利尔大学精神健康研究所研究中心）；ATR Computational Neuroscience Laboratories（ATR计算神经科学实验室）；University of Florida（佛罗里达大学）；University of California Los Angeles（加州大学洛杉矶分校）；RIKEN Center for Brain Science（理化学研究所脑科学中心）
Massachusetts Institute of Technology（麻省理工学院）；The Ohio State University（俄亥俄州立大学）
Identifying computational mechanisms for memorization and retrieval of data is a long-standing problem at the intersection of machine learning and neuroscience. Our main finding is that standard overparameterized deep neural networks trained using standard optimization methods implement such a mechanism for real-valued data. Empirically, we show that: (1) overparameterized autoencoders store training samples as attractors, and thus, iterating the learned map leads to sample recovery; (2) the same mechanism allows for encoding sequences of examples, and serves as an even more efficient mechanis…
This paper presents the largest publicly available, non-simulated, fleet-wide aircraft flight recording and maintenance log data for use in predicting part failure and maintenance need. We present 31,177 hours of flight data across 28,935 flights, which occur relative to 2,111 unplanned maintenance events clustered into 36 types of maintenance issues. Flights are annotated as before or after maintenance, with some flights occurring on the day of maintenance. Collecting data to evaluate predictive maintenance systems is challenging because it is difficult, dangerous, and unethical to generate d…
Material Research Institute, Queen Mary, University of London（伦敦大学玛丽女王学院材料研究所）
In the majority of molecular optimization tasks, predictive machine learning (ML) models are limited due to the unavailability and cost of generating big experimental datasets on the specific task. To circumvent this limitation, ML models are trained on big theoretical datasets or experimental indicators of molecular suitability that are either publicly available or inexpensive to acquire. These approaches produce a set of candidate molecules which have to be ranked using limited experimental data or expert knowledge. Under the assumption that structure is related to functionality, here we use…
To accommodate the unprecedented increase of commercial airlines over the next ten years, the Next Generation Air Transportation System (NextGen) has been implemented in the USA that records large-scale Air Traffic Management (ATM) data to make air travel safer, more efficient, and more economical. A key role of collaborative decision making for air traffic scheduling and airspace resource management is the accurate prediction of flight delay. There has been a lot of attempts to apply data-driven methods such as machine learning to forecast flight delay situation using air traffic data of depa…
With substantial recent developments in aviation technologies, Unmanned Aerial Vehicles (UAVs) are becoming increasingly integrated in commercial and military operations internationally. Research into the applications of aircraft data is essential in improving safety, reducing operational costs, and developing the next frontier of aerial technology. Having an outlier detection system that can accurately identify anomalous behaviour in aircraft is crucial for these reasons. This paper proposes a system incorporating a Long Short-Term Memory (LSTM) Deep Learning Autoencoder based method with a n…
Argonne National Laboratory（阿贡国家实验室）；Carnegie Mellon University（卡内基梅隆大学）
The goal of this work is to address two limitations in autoencoder-based models: latent space interpretability and compatibility with unstructured meshes. This is accomplished here with the development of a novel graph neural network (GNN) autoencoding architecture with demonstrations on complex fluid flow applications. To address the first goal of interpretability, the GNN autoencoder achieves reduction in the number nodes in the encoding stage through an adaptive graph reduction procedure. This reduction procedure essentially amounts to flowfield-conditioned node sampling and sensor identifi…
Deep Domain Adaptation for Turbofan Engine Remaining Useful Life Prediction: Methodologies, Evaluation and Future Trends
Institute for Infocomm Research, Agency for Science, Technology and Research, Singapore（新加坡资讯通信研究院，科技研究局）；Propulsion and Space Research Center, Technology Innovation Institute, UAE（阿联酋技术创新研究所推进与航天研究中心）；James Watt School of Engineering, University of Glasgow, UK（格拉斯哥大学詹姆斯·瓦特工程学院）；ISTD Pillar at SUTD（新加坡科技设计大学ISTD支柱）
Uncertainty Quantification in Machine Learning for Engineering Design and Health Prognostics: A Tutorial
Iowa State University（爱荷华州立大学）；ETH（苏黎世联邦理工学院）；University of Michigan（密歇根大学）；University of Michigan-Dearborn（密歇根大学迪尔伯恩分校）；EPFL（洛桑联邦理工学院）；Sandia National Laboratories（桑迪亚国家实验室）；Georgia Institute of Technology（佐治亚理工学院）；The Hong Kong Polytechnic University（香港理工大学）；University of Connecticut（康涅狄格大学）
Connecting the Dots: A Machine Learning Ready Dataset for Ionospheric Forecasting Models
Department of Information and Computer Science（信息与计算机科学系）；University of Hawai‘i at Mānoa（夏威夷大学毛纳罗亚分校）；Department of Engineering Science（工程科学系）；University of Oxford（牛津大学）；Università degli Studi di Roma Sapienza（罗马大学）；Free Flight Research Lab（自由飞行研究实验室）；University of New Hampshire（新罕布什尔大学）；European Space Agency (ESA)（欧洲航天局）；NASA Jet Propulsion Laboratory（美国宇航局喷气推进实验室）；NASA Headquarters（美国宇航局总部）；Space Weather Technology, Research, and Education Center（空间天气技术、研究与教育中心）；University of Colorado Boulder（科罗拉多大学博尔德分校）；Department of Computer Science（计算机科学系）；University of York & King’s College London（约克大学及伦敦国王学院）
