Determinantal Point Processes Implicitly Regularize Semi-parametric Regression Problems

Bayesian experimental design using regularized determinantal point processes

In experimental design, we are given $n$ vectors in $d$ dimensions, and our goal is to select $k\ll n$ of them to perform expensive measurements, e.g., to obtain labels/responses, for a linear regression task. Many statistical criteria have been proposed for choosing the optimal design, with popular choices including A- and D-optimality. If prior knowledge is given, typically in the form of a $d\times d$ precision matrix $\mathbf A$, then all of the criteria can be extended to incorporate that information via a Bayesian framework. In this paper, we demonstrate a new fundamental connection betw…

SGD Implicitly Regularizes Generalization Error

We derive a simple and model-independent formula for the change in the generalization gap due to a gradient descent update. We then compare the change in the test error for stochastic gradient descent to the change in test error from an equivalent number of gradient descent updates and show explicitly that stochastic gradient descent acts to regularize generalization error by decorrelating nearby updates. These calculations depends on the details of the model only through the mean and covariance of the gradient distribution, which may be readily measured for particular models of interest. We d…

Fast determinantal point processes via distortion-free intermediate sampling

Given a fixed $n\times d$ matrix $\mathbf{X}$, where $n\gg d$, we study the complexity of sampling from a distribution over all subsets of rows where the probability of a subset is proportional to the squared volume of the parallelepiped spanned by the rows (a.k.a. a determinantal point process). In this task, it is important to minimize the preprocessing cost of the procedure (performed once) as well as the sampling cost (performed repeatedly). To that end, we propose a new determinantal point process algorithm which has the following two properties, both of which are novel: (1) a preprocessi…

Class-incremental Learning using a Sequence of Partial Implicitly Regularized Classifiers

In class-incremental learning, the objective is to learn a number of classes sequentially without having access to the whole training data. However, due to a problem known as catastrophic forgetting, neural networks suffer substantial performance drop in such settings. The problem is often approached by experience replay, a method which stores a limited number of samples to be replayed in future steps to reduce forgetting of the learned classes. When using a pretrained network as a feature extractor, we show that instead of training a single classifier incrementally, it is better to train a nu…

用于回归的深度线性网络被隐式正则化趋向平坦极小值

Deep linear networks for regression are implicitly regularized towards flat minima

本文分析单变量回归深度线性网络的锐度，证明极小值锐度下界随深度线性增长，而梯度流在小规模与残差初始化下均将解隐式正则化到接近该下界的平坦极小值。

自然私有推荐：行列式点过程

Naturally Private Recommendations with Determinantal Point Processes

本文探讨行列式点过程在推荐中的自然隐私性，分析其满足ε-差分隐私所需的改动与敏感性，并提出更高效的替代方案以优化隐私-效用权衡。