RobustTrend: A Huber Loss with a Combined First and Second Order Difference Regularization for Time Series Trend Filtering
Alibaba Group（阿里巴巴集团）
Qingsong Wen, Jingkun Gao, Xiaomin Song, Liang Sun, Jian Tan
备注：
Accepted to the 28th International Joint Conference on Artificial Intelligence (IJCAI 2019), 7 pages. v2: added related references and adjusted font size in figures
Extracting the underlying trend signal is a crucial step to facilitate time series analysis like forecasting and anomaly detection. Besides noise signal, time series can contain not only outliers but also abrupt trend changes in real-world scenarios. To deal with these challenges, we propose a robust trend filtering algorithm based on robust statistics and sparse learning. Specifically, we adopt the Huber loss to suppress outliers, and utilize a combination of the first order and second order difference on the trend component as regularization to capture both slow and abrupt trend changes. Furthermore, an efficient method is designed to solve the proposed robust trend filtering based on majorization minimization (MM) and alternative direction method of multipliers (ADMM). We compared our proposed robust trend filter with other nine state-of-the-art trend filtering algorithms on both synthetic and real-world datasets. The experiments demonstrate that our algorithm outperforms existing methods.
Columbia University（哥伦比亚大学）；Zuckerman Institute（祖克曼研究所）
In this paper, we expand upon the theory of trend filtering by introducing the use of the Wasserstein metric as a means to control the amount of spatiotemporal variation in filtered time series data. While trend filtering utilizes regularization to produce signal estimates that are piecewise linear, in the case of $\ell_1$ regularization, or temporally smooth, in the case of $\ell_2$ regularization, it ignores the topology of the spatial distribution of signal. By incorporating the information about the underlying metric space of the pixel layout, the Wasserstein metric is an attractive choice…
University of California, Davis（加州大学戴维斯分校）；Amazon（亚马逊）
In this paper, we consider recommender systems with side information in the form of graphs. Existing collaborative filtering algorithms mainly utilize only immediate neighborhood information and have a hard time taking advantage of deeper neighborhoods beyond 1-2 hops. The main caveat of exploiting deeper graph information is the rapidly growing time and space complexity when incorporating information from these neighborhoods. In this paper, we propose using Graph DNA, a novel Deep Neighborhood Aware graph encoding algorithm, for exploiting deeper neighborhood information. DNA encoding compute…
Istituto Italiano di Tecnologia（意大利理工学院）；University of Genova（热那亚大学）；University of Verona（维罗纳大学）
This paper addresses the scalar regression problem through a novel solution to exactly optimize the Huber loss in a general semi-supervised setting, which combines multi-view learning and manifold regularization. We propose a principled algorithm to 1) avoid computationally expensive iterative schemes while 2) adapting the Huber loss threshold in a data-driven fashion and 3) actively balancing the use of labelled data to remove noisy or inconsistent annotations at the training stage. In a wide experimental evaluation, dealing with diverse applications, we assess the superiority of our paradigm…
University of Wisconsin-Madison（威斯康星大学麦迪逊分校）；University of Cambridge（剑桥大学）
We study the problem of linear regression where both covariates and responses are potentially (i) heavy-tailed and (ii) adversarially contaminated. Several computationally efficient estimators have been proposed for the simpler setting where the covariates are sub-Gaussian and uncontaminated; however, these estimators may fail when the covariates are either heavy-tailed or contain outliers. In this work, we show how to modify the Huber regression, least trimmed squares, and least absolute deviation estimators to obtain estimators which are simultaneously computationally and statistically effic…
A methodology to perform topological regularization via information filtering network is introduced. This methodology can be directly applied to covariance selection problem providing an instrument for sparse probabilistic modeling with both linear and non-linear multivariate probability distributions such as the elliptical and generalized hyperbolic families. It can also be directly implemented for $L_0$-norm regularized multicollinear regression. In this paper, I describe in detail an application to sparse modeling with multivariate Student-t. A specific $L_0$-norm regularized expectation-ma…
This paper addresses the problem of segmenting a time-series with respect to changes in the mean value or in the variance. The first case is when the time data is modeled as a sequence of independent and normal distributed random variables with unknown, possibly changing, mean value but fixed variance. The main assumption is that the mean value is piecewise constant in time, and the task is to estimate the change times and the mean values within the segments. The second case is when the mean value is constant, but the variance can change. The assumption is that the variance is piecewise consta…
In this paper, we present two localized graph filtering based methods for interpolating graph signals defined on the vertices of arbitrary graphs from only a partial set of samples. The first method is an extension of previous work on reconstructing bandlimited graph signals from partially observed samples. The iterative graph filtering approach very closely approximates the solution proposed in the that work, while being computationally more efficient. As an alternative, we propose a regularization based framework in which we define the cost of reconstruction to be a combination of smoothness…
This paper investigates the use of extended Kalman filtering to train recurrent neural networks with rather general convex loss functions and regularization terms on the network parameters, including $\ell_1$-regularization. We show that the learning method is competitive with respect to stochastic gradient descent in a nonlinear system identification benchmark and in training a linear system with binary outputs. We also explore the use of the algorithm in data-driven nonlinear model predictive control and its relation with disturbance models for offset-free closed-loop tracking.
University of California（加利福尼亚大学）；INRIA（法国国家信息与自动化研究所）；Laboratoire d’Informatique de l’Ecole Normale Supérieure（巴黎高等师范学院计算机科学实验室）；INSEAD（欧洲工商管理学院）；Mines ParisTech（巴黎高等矿业学院）；Institut Curie（居里研究所）；Inserm（法国国家健康与医学研究院）
We present a general approach for collaborative filtering (CF) using spectral regularization to learn linear operators from "users" to the "objects" they rate. Recent low-rank type matrix completion approaches to CF are shown to be special cases. However, unlike existing regularization based CF methods, our approach can be used to also incorporate information such as attributes of the users or the objects -- a limitation of existing regularization based CF methods. We then provide novel representer theorems that we use to develop new estimation methods. We provide learning algorithms based on …
Massachusetts Institute of Technology（麻省理工学院）；Toyota Technological Institute–Chicago（芝加哥丰田技术研究所）
We show that matrix completion with trace-norm regularization can be significantly hurt when entries of the matrix are sampled non-uniformly. We introduce a weighted version of the trace-norm regularizer that works well also with non-uniform sampling. Our experimental results demonstrate that the weighted trace-norm regularization indeed yields significant gains on the (highly non-uniformly sampled) Netflix dataset.
Regularization and normalization have become indispensable components in training deep neural networks, resulting in faster training and improved generalization performance. We propose the projected error function regularization loss (PER) that encourages activations to follow the standard normal distribution. PER randomly projects activations onto one-dimensional space and computes the regularization loss in the projected space. PER is similar to the Pseudo-Huber loss in the projected space, thus taking advantage of both $L^1$ and $L^2$ regularization losses. Besides, PER can capture the inte…
Feature selection is one of the most decisive tools in understanding data and machine learning models. Among other methods, sparsity induced by $L^{1}$ penalty is one of the simplest and best studied approaches to this problem. Although such regularization is frequently used in neural networks to achieve sparsity of weights or unit activations, it is unclear how it can be employed in the feature selection problem. This work aims at extending the neural network with ability to automatically select features by rethinking how the sparsity regularization can be used, namely, by stochastically pena…
Interpretable Operator Learning for Inverse Problems via Adaptive Spectral Filtering: Convergence and Discretization Invariance
School of Instrumentation Science and Engineering, Harbin Institute of Technology（仪器科学与工程学院，哈尔滨工业大学）；School of Mathematics, Jilin University（吉林大学数学学院）；School of Mathematics, Inner Mongolia University（内蒙古大学数学学院）
Southwest University of Science and Technology（西南科技大学）；PUC-Rio（里约热内卢天主教大学）；Xi’an Jiaotong University（西安交通大学）
In this paper, we propose a general robust subband adaptive filtering (GR-SAF) scheme against impulsive noise by minimizing the mean square deviation under the random-walk model with individual weight uncertainty. Specifically, by choosing different scaling factors such as from the M-estimate and maximum correntropy robust criteria in the GR-SAF scheme, we can easily obtain different GR-SAF algorithms. Importantly, the proposed GR-SAF algorithm can be reduced to a variable regularization robust normalized SAF algorithm, thus having fast convergence rate and low steady-state error. Simulations …
This paper proposes a novel model for the rating prediction task in recommender systems which significantly outperforms previous state-of-the art models on a time-split Netflix data set. Our model is based on deep autoencoder with 6 layers and is trained end-to-end without any layer-wise pre-training. We empirically demonstrate that: a) deep autoencoder models generalize much better than the shallow ones, b) non-linear activation functions with negative parts are crucial for training deep models, and c) heavy use of regularization techniques such as dropout is necessary to prevent over-fiting.…
Inria（法国国家信息与自动化研究所）；École normale supérieure（巴黎高等师范学院）；CNRS（法国国家科学研究中心）；PSL Research University（巴黎文理研究大学）；Univ. Grenoble Alpes（格勒诺布尔阿尔卑斯大学）；Grenoble INP（格勒诺布尔理工学院）
The theory of spectral filtering is a remarkable tool to understand the statistical properties of learning with kernels. For least squares, it allows to derive various regularization schemes that yield faster convergence rates of the excess risk than with Tikhonov regularization. This is typically achieved by leveraging classical assumptions called source and capacity conditions, which characterize the difficulty of the learning task. In order to understand estimators derived from other loss functions, Marteau-Ferey et al. have extended the theory of Tikhonov regularization to generalized self…
Huazhong University of Science and Technology（华中科技大学）
The electroencephalogram (EEG) is the most popular form of input for brain computer interfaces (BCIs). However, it can be easily contaminated by various artifacts and noise, e.g., eye blink, muscle activities, powerline noise, etc. Therefore, the EEG signals are often filtered both spatially and temporally to increase the signal-to-noise ratio before they are fed into a machine learning algorithm for recognition. This paper considers spatial filtering, particularly, the common spatial pattern (CSP) filters for EEG classification. In binary classification, CSP seeks a set of filters to maximize…
This paper describes the solution method taken by LeBuSiShu team for track1 in ACM KDD CUP 2011 contest (resulting in the 5th place). We identified two main challenges: the unique item taxonomy characteristics as well as the large data set size.To handle the item taxonomy, we present a novel method called Matrix Factorization Item Taxonomy Regularization (MFITR). MFITR obtained the 2nd best prediction result out of more then ten implemented algorithms. For rapidly computing multiple solutions of various algorithms, we have implemented an open source parallel collaborative filtering library on …
LITIS EA 4108, INSA-Université de Rouen（鲁昂国立应用科学学院-鲁昂大学 LITIS EA 4108实验室）
We address in this paper the problem of multi-channel signal sequence labeling. In particular, we consider the problem where the signals are contaminated by noise or may present some dephasing with respect to their labels. For that, we propose to jointly learn a SVM sample classifier with a temporal filtering of the channels. This will lead to a large margin filtering that is adapted to the specificity of each channel (noise and time-lag). We derive algorithms to solve the optimization problem and we discuss different filter regularizations for automated scaling or selection of channels. Our a…
We consider online linear optimization over symmetric positive semi-definite matrices, which has various applications including the online collaborative filtering. The problem is formulated as a repeated game between the algorithm and the adversary, where in each round t the algorithm and the adversary choose matrices X_t and L_t, respectively, and then the algorithm suffers a loss given by the Frobenius inner product of X_t and L_t. The goal of the algorithm is to minimize the cumulative loss. We can employ a standard framework called Follow the Regularized Leader (FTRL) for designing algorit…
A regularized optimization problem over a large unstructured graph is studied, where the regularization term is tied to the graph geometry. Typical regularization examples include the total variation and the Laplacian regularizations over the graph. When applying the proximal gradient algorithm to solve this problem, there exist quite affordable methods to implement the proximity operator (backward step) in the special case where the graph is a simple path without loops. In this paper, an algorithm, referred to as "Snake", is proposed to solve such regularized problems over general graphs, by …
Harbin Institute of Technology, Shenzhen（哈尔滨工业大学（深圳））
The recommender systems have long been investigated in the literature. Recently, users' implicit feedback like `click' or `browse' are considered to be able to enhance the recommendation performance. Therefore, a number of attempts have been made to resolve this issue. Among them, the variational autoencoders (VAE) approach already achieves a superior performance. However, the distributions of the encoded latent variables overlap a lot which may restrict its recommendation ability. To cope with this challenge, this paper tries to extend the Wasserstein autoencoders (WAE) for collaborative filt…
Nonnegative Tucker Factorization (NTF) minimizes the euclidean distance or Kullback-Leibler divergence between the original data and its low-rank approximation which often suffers from grossly corruptions or outliers and the neglect of manifold structures of data. In particular, NTF suffers from rotational ambiguity, whose solutions with and without rotation transformations are equally in the sense of yielding the maximum likelihood. In this paper, we propose three Robust Manifold NTF algorithms to handle outliers by incorporating structural knowledge about the outliers. They first applies a h…
