Strong Asymptotic Assertions for Discrete MDL in Regression and Classification

Asymptotics of Discrete MDL for Online Prediction

Minimum Description Length (MDL) is an important principle for induction and prediction, with strong relations to optimal Bayesian learning. This paper deals with learning non-i.i.d. processes by means of two-part MDL, where the underlying model class is countable. We consider the online learning framework, i.e. observations come in one by one, and the predictor is allowed to update his state of mind after each time step. We identify two ways of predicting by MDL for this setup, namely a static} and a dynamic one. (A third variant, hybrid MDL, will turn out inferior.) We will prove that under …

重访过参数化模型中的最小描述长度复杂度

Revisiting minimum description length complexity in overparameterized models

本文提出一种新的基于最小描述长度原则的复杂度度量MDL-COMP，适用于过参数化模型，理论分析表明其与参数数量、奇异值和信噪比相关，并可用于指导岭回归的超参数调整，优于交叉验证且节省计算成本。

The Loss Rank Principle for Model Selection

We introduce a new principle for model selection in regression and classification. Many regression models are controlled by some smoothness or flexibility or complexity parameter c, e.g. the number of neighbors to be averaged over in k nearest neighbor (kNN) regression or the polynomial degree in regression with polynomials. Let f_D^c be the (best) regressor of complexity c on data D. A more flexible regressor can fit more data D' well than a more rigid one. If something (here small loss) is easy to achieve it's typically worth less. We define the loss rank of f_D^c as the number of other (fic…