具有最佳缩放定律的图灵泛学习器
Turing-Universal Learners with Optimal Scaling Laws
摘要:
对于给定的分布、学习算法和性能度量,收敛速率(或数据缩放定律)是作为训练样本数量的函数的算法的测试性能的渐近行为。理论和实践中的许多学习方法都具有幂律速率,即对于某些$α> 0$,性能尺度为$n^{-α}$。此外,理论家和实践者都关心在感兴趣的设置下提高他们的学习算法的速率。我们观察到存在一个“通用学习器”,它在指定的运行时间(例如$O(n^2)$)内实现了所有学习算法中最好的可能的分布相关渐近速率,同时在此运行时间内只会引起多对数减速。该算法是均匀的,并且不依赖于分布,并且对于所有分布都实现了最佳可能的速率。这个构造本身是Levin的通用搜索的简单扩展(Levin,1973)。就像通用搜索一样,通用学习器根本不是实用的,主要是理论和哲学上的兴趣。
英文摘要:
For a given distribution, learning algorithm, and performance metric, the rate of convergence (or data-scaling law) is the asymptotic behavior of the algorithm's test performance as a function of number of train samples. Many learning methods in both theory and practice have power-law rates, i.e. performance scales as $n^{-α}$ for some $α> 0$. Moreover, both theoreticians and practitioners are concerned with improving the rates of their learning algorithms under settings of interest. We observe the existence of a "universal learner", which achieves the best possible distribution-dependent asymptotic rate among all learning algorithms within a specified runtime (e.g. $O(n^2)$), while incurring only polylogarithmic slowdown over this runtime. This algorithm is uniform, and does not depend on the distribution, and yet achieves best-possible rates for all distributions. The construction itself is a simple extension of Levin's universal search (Levin, 1973). And much like universal search, the universal learner is not at all practical, and is primarily of theoretical and philosophical interest.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。