The No Free Lunch theorems are often used to argue that domain specific knowledge is required to design successful algorithms. We use algorithmic information theory to argue the case for a universal bias allowing an algorithm to succeed in all interesting problem domains. Additionally, we give a new algorithm for off-line classification, inspired by Solomonoff induction, with good performance on all structured problems under reasonable assumptions. This includes a proof of the efficacy of the well-known heuristic of randomly selecting training data in the hope of reducing misclassification rates.
We provide a new representation-independent formulation of Occam's razor theorem, based on Kolmogorov complexity. This new formulation allows us to: (i) Obtain better sample complexity than both length-based and VC-based versions of Occam's razor theorem, in many applications. (ii) Achieve a sharper reverse of Occam's razor theorem than previous work. Specifically, we weaken the assumptions made in an earlier publication, and extend the reverse to superpolynomial running times.
In over-parameterized deep neural networks there can be many possible parameter configurations that fit the training data exactly. However, the properties of these interpolating solutions are poorly understood. We argue that over-parameterized neural networks trained with stochastic gradient descent are subject to a Geometric Occam's Razor; that is, these networks are implicitly regularized by the geometric model complexity. For one-dimensional regression, the geometric model complexity is simply given by the arc length of the function. For higher-dimensional settings, the geometric model comp…
Deep neural networks have an inbuilt Occam's razor
Rudolf Peierls Centre for Theoretical Physics, University of Oxford（鲁道夫·皮埃尔斯理论物理中心，牛津大学）；Physical and Theoretical Chemistry Laboratory, University of Oxford（物理与理论化学实验室，牛津大学）
Achieving Occam's Razor: Deep Learning for Optimal Model Reduction
State University of New York at Stony Brook（纽约州立大学石溪分校）；Laufer Center for Physical and Quantitative Biology（劳弗物理与定量生物学中心）；Memorial Sloan Kettering Cancer Center（纪念斯隆·凯特琳癌症中心）
Shaving Weights with Occam's Razor: Bayesian Sparsification for Neural Networks Using the Marginal Likelihood
Technical University of Munich（慕尼黑工业大学）；Munich Center for Machine Learning（慕尼黑机器学习中心）；ETH Zürich（苏黎世联邦理工学院）；Max Planck Institute for Intelligent Systems, Tübingen（马克斯·普朗克智能系统研究所（蒂宾根））；Pruna AI（Pruna AI公司）；Helmholtz AI, Munich（亥姆霍兹人工智能研究所（慕尼黑））
Centre National de la Recherche Scientifique（法国国家科学研究中心）；CNRS-Strasbourg University（法国国家科学研究中心-斯特拉斯堡大学）；NTNU Trondheim（特隆赫姆挪威科技大学）；Institute of Applied Informatics（应用信息学研究所）；University of South Bohemia Czech Republic（捷克南波希米亚大学）；Kogakkan University（皇学馆大学）；University of Applied Sciences Upper Austria（上奥地利应用科学大学）；Namics—A Merkle Company（Namics——Merkle旗下公司）
Detecting quality in large unstructured datasets requires capacities far beyond the limits of human perception and communicability and, as a result, there is an emerging trend towards increasingly complex analytic solutions in data science to cope with this problem. This new trend towards analytic complexity represents a severe challenge for the principle of parsimony or Occams Razor in science. This review article combines insight from various domains such as physics, computational science, data engineering, and cognitive science to review the specific properties of big data. Problems for det…
Federal University of Minas Gerais（米纳斯吉拉斯联邦大学）；The University of Newcastle（纽卡斯尔大学）；Uppsala University（乌普萨拉大学）
System identification aims to build models of dynamical systems from data. Traditionally, choosing the model requires the designer to balance between two goals of conflicting nature; the model must be rich enough to capture the system dynamics, but not so flexible that it learns spurious random effects from the dataset. It is typically observed that the model validation performance follows a U-shaped curve as the model complexity increases. Recent developments in machine learning and statistics, however, have observed situations where a "double-descent" curve subsumes this U-shaped model-perfo…
Xi’an Jiaotong University（西安交通大学）；Singapore Management University（新加坡管理大学）；Harbin Institute of Technology（哈尔滨工业大学）；Pengcheng Laboratory（鹏城实验室）
State-of-the-art deep neural networks are still struggling to address the catastrophic forgetting problem in continual learning. In this paper, we propose one simple paradigm (named as S-Prompting) and two concrete approaches to highly reduce the forgetting degree in one of the most typical continual learning scenarios, i.e., domain increment learning (DIL). The key idea of the paradigm is to learn prompts independently across domains with pre-trained transformers, avoiding the use of exemplars that commonly appear in conventional methods. This results in a win-win game where the prompting can…
OScaR: The Occam's Razor for Extreme KV Cache Quantization in LLMs and Beyond
Tsinghua University（清华大学）；Meituan LongCat Team（美团LongCat团队）；The University of Hong Kong（香港大学）；The University of Edinburgh（爱丁堡大学）；UCAS（中国科学技术大学）；The Hong Kong Polytechnic University（香港理工大学）
University of Helsinki（赫尔辛基大学）；University of Oslo（奥斯陆大学）；Wellcome Sanger Institute（威康桑格研究所）；KTH Royal Institute of Technology（KTH皇家理工学院）
Choice of appropriate structure and parametric dimension of a model in the light of data has a rich history in statistical research, where the first seminal approaches were developed in 1970s, such as the Akaike's and Schwarz's model scoring criteria that were inspired by information theory and embodied the rationale called Occam's razor. After those pioneering works, model choice was quickly established as its own field of research, gaining considerable attention in both computer science and statistics. However, to date, there have been limited attempts to derive scoring criteria for simulato…
University of Chinese Academy of Sciences（中国科学院大学）
Transfer learning aims at transferring knowledge from a well-labeled domain to a similar but different domain with limited or no labels. Unfortunately, existing learning-based methods often involve intensive model selection and hyperparameter tuning to obtain good results. Moreover, cross-validation is not possible for tuning hyperparameters since there are often no labels in the target domain. This would restrict wide applicability of transfer learning especially in computationally-constraint devices such as wearables. In this paper, we propose a practically Easy Transfer Learning (EasyTL) ap…
Tel Aviv University（特拉维夫大学）；Facebook AI Research（脸书人工智能研究院）
The recent empirical success of unsupervised cross-domain mapping algorithms, between two domains that share common characteristics, is not well-supported by theoretical justifications. This lacuna is especially troubling, given the clear ambiguity in such mappings. We work with adversarial training methods based on IPMs and derive a novel risk bound, which upper bounds the risk between the learned mapping $h$ and the target mapping $y$, by a sum of three terms: (i) the risk between $h$ and the most distant alternative mapping that was learned by the same cross-domain mapping algorithm, (ii) t…
