The problem of missing data, usually absent incurated and competition-standard datasets, is an unfortunate reality for most machine learning models used in industry applications. Recent work has focused on understanding the nature and the negative effects of such phenomena, while devising solutions for optimal imputation of the missing data, using both discriminative and generative approaches. We propose a novel mechanism based on multi-head attention which can be applied effortlessly in any model and achieves better downstream performance without the introduction of the full dataset in any part of the modeling pipeline. Our method inductively models patterns of missingness in the input data in order to increase the performance of the downstream task. Finally, after evaluating our method against baselines for a number of datasets, we found performance gains that tend to be larger in scenarios of high missingness.
Graduate School of Informatics, Kyoto University（京都大学信息学研究生院）
We investigate a monotonic multihead attention (MMA) by extending hard monotonic attention to Transformer-based automatic speech recognition (ASR) for online streaming applications. For streaming inference, all monotonic attention (MA) heads should learn proper alignments because the next token is not generated until all heads detect the corresponding token boundaries. However, we found not all MA heads learn alignments with a naïve implementation. To encourage every head to learn alignments properly, we propose HeadDrop regularization by masking out a part of heads stochastically during train…
SPAT: Sensitivity-based Multihead-attention Pruning on Time Series Forecasting Models
School of Artificial Intelligence, Nanjing University, China（人工智能学院，南京大学）；National Key Laboratory for Novel Software Technology, Nanjing University, China（新型软件技术国家重点实验室，南京大学）；School of Electronic Science and Engineering, Nanjing University, China（电子科学与工程学院，南京大学）
本文提出 SPAT 结构化剪枝方法，利用动态 SEND 指标识别并移除整个冗余注意力模块，在多元时间序列预测中降低误差与 FLOPs，并超越多种 SOTA 方法。
D3MES: Diffusion Transformer with multihead equivariant self-attention for 3D molecule generation
School of Physics and Electronic Engineering, Jiangsu University（江苏大学物理与电子工程学院）；Jiangsu Engineering Research Center on Quantum Perception and Intelligent Detection of Agricultural Information（江苏省农业信息量子感知与智能检测工程研究中心）
Temporally Multi-Scale Sparse Self-Attention for Physical Activity Data Imputation
University of Massachusetts Amherst（马萨诸塞大学阿默斯特分校）；Georgia Institute of Technology（佐治亚理工学院）；University of Illinois Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）；University of Memphis（孟菲斯大学）
RMAU-NET: A Residual-Multihead-Attention U-Net Architecture for Landslide Segmentation and Detection from Remote Sensing Images
Van Lang University（范兰大学）；Austrian Institute of Technology（奥地利技术研究院）；Troyes University（特鲁瓦大学）；HCM University of Technology（胡志明技术大学）；Ton Duc Thang University（-ton Duc Thang大学）
本文提出了一种基于残差多头注意力 U-Net 架构的端到端深度学习模型 RMAU-NET，利用遥感图像进行滑坡检测与分割，在多个基准数据集上取得了优异的 F1 和 mIoU 分数，证明了其在实际滑坡观测系统中的应用潜力。
ST-GIN: An Uncertainty Quantification Approach in Traffic Data Imputation with Spatio-temporal Graph Attention and Bidirectional Recurrent United Neural Networks
University of Pennsylvania（宾夕法尼亚大学）；Massachusetts Institute of Technology（麻省理工学院）；University of Florida（佛罗里达大学）；Wuhan University（武汉大学）；Duke Kunshan University（昆山杜克大学）
Missing data is a recurrent and challenging problem, especially when using machine learning algorithms for real-world applications. For this reason, missing data imputation has become an active research area, in which recent deep learning approaches have achieved state-of-the-art results. We propose DAEMA (Denoising Autoencoder with Mask Attention), an algorithm based on a denoising autoencoder architecture with an attention mechanism. While most imputation algorithms use incomplete inputs as they would use complete data - up to basic preprocessing (e.g. mean imputation) - DAEMA leverages a ma…
Most practical data science problems encounter missing data. A wide variety of solutions exist, each with strengths and weaknesses that depend upon the missingness-generating process. Here we develop a theoretical framework for training and inference using only observed variables enabling modeling of incomplete datasets without imputation. Using an information and measure-theoretic argument we construct models with latent space representations that regularize against the potential bias introduced by missing data. The theoretical properties of this approach are demonstrated empirically using a …
Time series data are ubiquitous in real-world applications. However, one of the most common problems is that the time series data could have missing values by the inherent nature of the data collection process. So imputing missing values from multivariate (correlated) time series data is imperative to improve a prediction performance while making an accurate data-driven decision. Conventional works for imputation simply delete missing values or fill them based on mean/zero. Although recent works based on deep neural networks have shown remarkable results, they still have a limitation to captur…
RMIT University（皇家墨尔本理工大学）；Cuebiq Inc.（Cuebiq公司）；University of New South Wales（新南威尔士大学）
Sparsity is a common issue in many trajectory datasets, including human mobility data. This issue frequently brings more difficulty to relevant learning tasks, such as trajectory imputation and prediction. Nowadays, little existing work simultaneously deals with imputation and prediction on human trajectories. This work plans to explore whether the learning process of imputation and prediction could benefit from each other to achieve better outcomes. And the question will be answered by studying the coexistence patterns between missing points and observed ones in incomplete trajectories. More …
Beihang University（北京航空航天大学）；University of Central Florida（中佛罗里达大学）
Spatiotemporal data mining plays an important role in air quality monitoring, crowd flow modeling, and climate forecasting. However, the originally collected spatiotemporal data in real-world scenarios is usually incomplete due to sensor failures or transmission loss. Spatiotemporal imputation aims to fill the missing values according to the observed values and the underlying spatiotemporal dependence of them. The previous dominant models impute missing values autoregressively and suffer from the problem of error accumulation. As emerging powerful generative models, the diffusion probabilistic…
STAMImputer: Spatio-Temporal Attention MoE for Traffic Data Imputation
School of Cyber Science and Technology, Beihang University（北航信息科学与技术学院）；Hangzhou Innovation Institute of BUAA, Hangzhou, China（北京航空航天大学杭州创新研究院）；Department of Computer Science and Technology, Shantou University（汕头大学计算机科学与技术学院）；School of Computer Science and Engineering, Central South University（中南大学计算机科学与工程学院）；Department of Computing, Macquarie University（麦考瑞大学计算机学院）；National University of Defense Technology, Changsha, China（国防科技大学）
Joint Learning of Unsupervised Multi-view Feature and Instance Co-selection with Cross-view Imputation
Joint Laboratory of Data Science and Business Intelligence, School of Statistics and Data Science, Southwestern University of Finance and Economics（数据科学与商务智能联合实验室，统计与数据科学学院，西南财经大学）；Department of Electrical Engineering and Computer Science, University of Kansas（电气工程与计算机科学系，堪萨斯大学）；School of Computing and Artificial Intelligence, Southwest Jiaotong University（计算与人工智能学院，西南交通大学）；Faculty of Mathematics and Computer Science, University of Münster（数学与计算机科学学院，穆斯堡大学）
The University of Tokyo（东京大学）；Preferred Networks（首选网络）
Missing value imputation in machine learning is the task of estimating the missing values in the dataset accurately using available information. In this task, several deep generative modeling methods have been proposed and demonstrated their usefulness, e.g., generative adversarial imputation networks. Recently, diffusion models have gained popularity because of their effectiveness in the generative modeling task in images, texts, audio, etc. To our knowledge, less attention has been paid to the investigation of the effectiveness of diffusion models for missing value imputation in tabular data…
Time series are all around in real-world applications. However, unexpected accidents for example broken sensors or missing of the signals will cause missing values in time series, making the data hard to be utilized. It then does harm to the downstream applications such as traditional classification or regression, sequential data integration and forecasting tasks, thus raising the demand for data imputation. Currently, time series data imputation is a well-studied problem with different categories of methods. However, these works rarely take the temporal relations among the observations and tr…
ETH Zurich（苏黎世联邦理工学院）；SIB Swiss Institute of Bioinformatics（SIB瑞士生物信息学研究所）
The signature transform is a 'universal nonlinearity' on the space of continuous vector-valued paths, and has received attention for use in machine learning on time series. However, real-world temporal data is typically observed at discrete points in time, and must first be transformed into a continuous path before signature techniques can be applied. We make this step explicit by characterising it as an imputation problem, and empirically assess the impact of various imputation strategies when applying signature-based neural nets to irregular time series data. For one of these strategies, Gau…
Efficient data augmentation using graph imputation neural networks
Department of Information Engineering, Electronics；Telecommunications (DIET), ``Sapienza'' University of Rome, Via Eudossiana 18, 00184, Rome. \ : \ simone.scardapane
Boundary-enhanced time series data imputation with long-term dependency diffusion models
Henan University（河南大学）；Yangtze Delta Region Institute (Quzhou), University of Electronic Science and Technology of China（电子科技大学长三角研究院（衢州））；University College London（伦敦大学学院）
