arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2402.03254stat.MLcs.ITcs.LGmath.IT
收藏

表示学习的最小描述长度与泛化保证

Minimum Description Length and Generalization Guarantees for Representation Learning

Paris Research Center, Huawei Technologies France(法国华为技术有限公司巴黎研发中心);Université Gustave Eiffel(古斯塔夫·埃菲尔大学)

Milad Sefidgaran, Abdellatif Zaidi, Piotr Krasnowski

AI总结:

本文提出基于最小描述长度的可压缩性框架,为表示学习算法推导出新的泛化误差上界,并引入数据依赖先验,优于经典先验。

摘要:

设计高效的统计监督学习算法的一个主要挑战是找到不仅在可用训练样本上表现良好,而且对未见数据也表现良好的表示。尽管表示学习的研究引起了广泛关注,但大多数现有方法都是启发式的;关于理论泛化保证知之甚少。在本文中,我们建立了一个可压缩性框架,该框架使我们能够根据标签或潜在变量(表示)的“最小描述长度”(MDL)推导出表示学习算法泛化误差的上界。我们的新界涉及训练集和测试集的表示(或标签)分布与固定先验之间的“多字母”相对熵,而不是编码器输入与表示之间的互信息,后者在相关文献中常被认为反映了算法的泛化能力,但实际上并不能做到这一点。特别是,这些新界反映了编码器的结构,并且对于确定性算法并非空洞的。我们的可压缩性方法本质上是信息论的,建立在Blum-Langford的PAC-MDL界之上,并引入了两个基本要素:块编码和有损压缩。后者使我们的方法将所谓的几何可压缩性作为特例包含在内。据作者所知,所建立的泛化界是首个针对信息瓶颈(IB)类型编码器和表示学习的此类界。最后,我们通过引入一种新的数据依赖先验部分利用了理论结果。数值模拟说明了精心选择的此类先验相对于IB中使用的经典先验的优势。

英文摘要:

A major challenge in designing efficient statistical supervised learning algorithms is finding representations that perform well not only on available training samples but also on unseen data. While the study of representation learning has spurred much interest, most existing such approaches are heuristic; and very little is known about theoretical generalization guarantees. In this paper, we establish a compressibility framework that allows us to derive upper bounds on the generalization error of a representation learning algorithm in terms of the "Minimum Description Length" (MDL) of the labels or the latent variables (representations). Rather than the mutual information between the encoder's input and the representation, which is often believed to reflect the algorithm's generalization capability in the related literature but in fact, falls short of doing so, our new bounds involve the "multi-letter" relative entropy between the distribution of the representations (or labels) of the training and test sets and a fixed prior. In particular, these new bounds reflect the structure of the encoder and are not vacuous for deterministic algorithms. Our compressibility approach, which is information-theoretic in nature, builds upon that of Blum-Langford for PAC-MDL bounds and introduces two essential ingredients: block-coding and lossy-compression. The latter allows our approach to subsume the so-called geometrical compressibility as a special case. To the best knowledge of the authors, the established generalization bounds are the first of their kind for Information Bottleneck (IB) type encoders and representation learning. Finally, we partly exploit the theoretical results by introducing a new data-dependent prior. Numerical simulations illustrate the advantages of well-chosen such priors over classical priors used in IB.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑