System identification aims to build models of dynamical systems from data. Traditionally, choosing the model requires the designer to balance between two goals of conflicting nature; the model must be rich enough to capture the system dynamics, but not so flexible that it learns spurious random effects from the dataset. It is typically observed that the model validation performance follows a U-shaped curve as the model complexity increases. Recent developments in machine learning and statistics, however, have observed situations where a "double-descent" curve subsumes this U-shaped model-performance curve. With a second decrease in performance occurring beyond the point where the model has reached the capacity of interpolating - i.e., (near) perfectly fitting - the training data. To the best of our knowledge, such phenomena have not been studied within the context of dynamic systems. The present paper aims to answer the question: "Can such a phenomenon also be observed when estimating parameters of dynamic systems?" We show that the answer is yes, verifying such behavior experimentally both for artificially generated and real-world datasets.
In over-parameterized deep neural networks there can be many possible parameter configurations that fit the training data exactly. However, the properties of these interpolating solutions are poorly understood. We argue that over-parameterized neural networks trained with stochastic gradient descent are subject to a Geometric Occam's Razor; that is, these networks are implicitly regularized by the geometric model complexity. For one-dimensional regression, the geometric model complexity is simply given by the arc length of the function. For higher-dimensional settings, the geometric model comp…
We provide a new representation-independent formulation of Occam's razor theorem, based on Kolmogorov complexity. This new formulation allows us to: (i) Obtain better sample complexity than both length-based and VC-based versions of Occam's razor theorem, in many applications. (ii) Achieve a sharper reverse of Occam's razor theorem than previous work. Specifically, we weaken the assumptions made in an earlier publication, and extend the reverse to superpolynomial running times.
Australian National University（澳大利亚国立大学）；ETH Zürich（苏黎世联邦理工学院）；NICTA（澳大利亚国家信息与通信技术研究院）
The No Free Lunch theorems are often used to argue that domain specific knowledge is required to design successful algorithms. We use algorithmic information theory to argue the case for a universal bias allowing an algorithm to succeed in all interesting problem domains. Additionally, we give a new algorithm for off-line classification, inspired by Solomonoff induction, with good performance on all structured problems under reasonable assumptions. This includes a proof of the efficacy of the well-known heuristic of randomly selecting training data in the hope of reducing misclassification rat…
Deep neural networks have an inbuilt Occam's razor
Rudolf Peierls Centre for Theoretical Physics, University of Oxford（鲁道夫·皮埃尔斯理论物理中心，牛津大学）；Physical and Theoretical Chemistry Laboratory, University of Oxford（物理与理论化学实验室，牛津大学）
Shaving Weights with Occam's Razor: Bayesian Sparsification for Neural Networks Using the Marginal Likelihood
Technical University of Munich（慕尼黑工业大学）；Munich Center for Machine Learning（慕尼黑机器学习中心）；ETH Zürich（苏黎世联邦理工学院）；Max Planck Institute for Intelligent Systems, Tübingen（马克斯·普朗克智能系统研究所（蒂宾根））；Pruna AI（Pruna AI公司）；Helmholtz AI, Munich（亥姆霍兹人工智能研究所（慕尼黑））
Achieving Occam's Razor: Deep Learning for Optimal Model Reduction
State University of New York at Stony Brook（纽约州立大学石溪分校）；Laufer Center for Physical and Quantitative Biology（劳弗物理与定量生物学中心）；Memorial Sloan Kettering Cancer Center（纪念斯隆·凯特琳癌症中心）
Centre National de la Recherche Scientifique（法国国家科学研究中心）；CNRS-Strasbourg University（法国国家科学研究中心-斯特拉斯堡大学）；NTNU Trondheim（特隆赫姆挪威科技大学）；Institute of Applied Informatics（应用信息学研究所）；University of South Bohemia Czech Republic（捷克南波希米亚大学）；Kogakkan University（皇学馆大学）；University of Applied Sciences Upper Austria（上奥地利应用科学大学）；Namics—A Merkle Company（Namics——Merkle旗下公司）
Detecting quality in large unstructured datasets requires capacities far beyond the limits of human perception and communicability and, as a result, there is an emerging trend towards increasingly complex analytic solutions in data science to cope with this problem. This new trend towards analytic complexity represents a severe challenge for the principle of parsimony or Occams Razor in science. This review article combines insight from various domains such as physics, computational science, data engineering, and cognitive science to review the specific properties of big data. Problems for det…
Xi’an Jiaotong University（西安交通大学）；Singapore Management University（新加坡管理大学）；Harbin Institute of Technology（哈尔滨工业大学）；Pengcheng Laboratory（鹏城实验室）
State-of-the-art deep neural networks are still struggling to address the catastrophic forgetting problem in continual learning. In this paper, we propose one simple paradigm (named as S-Prompting) and two concrete approaches to highly reduce the forgetting degree in one of the most typical continual learning scenarios, i.e., domain increment learning (DIL). The key idea of the paradigm is to learn prompts independently across domains with pre-trained transformers, avoiding the use of exemplars that commonly appear in conventional methods. This results in a win-win game where the prompting can…
This article applies the principle of Occam's Razor to non-parametric model building of statistical data, by finding a model with the minimal number of bits, leading to an exceptionally effective regularization method for probability density estimators. The idea comes from the fact that likelihood maximization also minimizes the number of bits required to encode a dataset. However, traditional methods overlook that the optimization of model parameters may also inadvertently play the part in encoding data points. The article shows how to extend the bit counting to the model parameters as well, …
OScaR: The Occam's Razor for Extreme KV Cache Quantization in LLMs and Beyond
Tsinghua University（清华大学）；Meituan LongCat Team（美团LongCat团队）；The University of Hong Kong（香港大学）；The University of Edinburgh（爱丁堡大学）；UCAS（中国科学技术大学）；The Hong Kong Polytechnic University（香港理工大学）
The University of Sydney（悉尼大学）；National Institute of Informatics（国立信息学研究所）
The appearance of the double-descent risk phenomenon has received growing interest in the machine learning and statistics community, as it challenges well-understood notions behind the U-shaped train-test curves. Motivated through Rissanen's minimum description length (MDL), Balasubramanian's Occam's Razor, and Amari's information geometry, we investigate how the logarithm of the model volume: $\log V$, works to extend intuition behind the AIC and BIC model selection criteria. We find that for the particular model classes of isotropic linear regression and statistical lattices, the $\log V$ te…
Booth School of Business University of Chicago（芝加哥大学商学院）；Volgenau School of Engineering, George Mason University（乔治·马歇尔大学工程学院）；Department of Systems Engineering and Operations Research George Mason University（乔治·马歇尔大学系统工程与运营管理系）
