arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2103.02165stat.MLcs.ITcs.LGmath.IT
收藏

简约推理

Parsimonious Inference

Sandia National Laboratories(桑迪亚国家实验室)

Jed A. Duersch, Thomas A. Catanach

摘要:

贝叶斯推断提供了一种独特的严格的方法来获得预测中的不确定性的原则性理由,但在机器学习上下文中,很难适当地表达一般的先验信念,其中计算架构是纯抽象的,需要实践者频繁修改以改善结果。简约推理是一种信息论的推理公式,它形式化了奥卡姆剃刀;我们更喜欢简单而充分的解释。我们的通用hyperprior分配的先验描述,编码为符号序列,通过扩展程序长度,Kolmogorov复杂性和Solomonoff的算法概率之间的核心关系。然后,我们将学习转换为信息最小化,当指定架构时,我们的复合信念变化,观察训练数据,并推断模型参数。通过区分模型复杂性和预测信息,我们的框架还量化了记忆现象。虽然我们的理论是通用的,但当数据集有限时,例如小或倾斜时,它是最关键的。我们开发了新的算法多项式回归和随机森林,适合于这样的数据,我们的实验证明。我们的方法将有效的编码与谨慎的采样策略相结合,在没有交叉验证的情况下构建预测集合,从而解决了如何有效地从数据中获得预测的根本挑战。

英文摘要:

Bayesian inference provides a uniquely rigorous approach to obtain principled justification for uncertainty in predictions, yet it is difficult to articulate suitably general prior belief in the machine learning context, where computational architectures are pure abstractions subject to frequent modifications by practitioners attempting to improve results. Parsimonious inference is an information-theoretic formulation of inference over arbitrary architectures that formalizes Occam's Razor; we prefer simple and sufficient explanations. Our universal hyperprior assigns plausibility to prior descriptions, encoded as sequences of symbols, by expanding on the core relationships between program length, Kolmogorov complexity, and Solomonoff's algorithmic probability. We then cast learning as information minimization over our composite change in belief when an architecture is specified, training data are observed, and model parameters are inferred. By distinguishing model complexity from prediction information, our framework also quantifies the phenomenon of memorization. Although our theory is general, it is most critical when datasets are limited, e.g. small or skewed. We develop novel algorithms for polynomial regression and random forests that are suitable for such data, as demonstrated by our experiments. Our approaches combine efficient encodings with prudent sampling strategies to construct predictive ensembles without cross-validation, thus addressing a fundamental challenge in how to efficiently obtain predictions from data.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑