arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 1711.07099stat.MLcs.LG
收藏

基于压缩的正则化及其在多任务学习中的应用

Compression-Based Regularization with an Application to Multi-Task Learning

Facultad de Ingeniería - Universidad de Buenos Aires(布宜诺斯艾利斯大学工程学院);CSC-CONICET(阿根廷国家科学与技术研究理事会计算科学中心);CentraleSupélec(巴黎中央理工-高等电力学院);CNRS(法国国家科学研究中心);Université Paris-Sud(巴黎南大学)

Matías Vera, Leonardo Rey Vega, Pablo Piantanida

摘要:

本文从信息论的角度研究了一个基于以下原则的学习问题:给定数据集中的任何规律性都可以用来从数据中提取紧凑特征,即,使用比完全描述数据本身所需的更少的位,以便构建相关内容的有意义的表示(多个标签)。我们首先介绍了噪声有损源编码范式与对数损失保真度标准,它提供了基本的权衡之间的\emph{cross-entropy loss}(平均风险)和信息率的功能(模型复杂性)。我们的方法允许信息理论制定的\emph{multi-task learning}(MTL)的问题,这是一个监督的学习框架中,几个相关的任务的预测模型共同学习,从共同的表示,以实现更好的泛化性能。然后,我们提出了一个迭代算法计算的最佳权衡和它的全局收敛性证明了一些条件成立。该算法的一个重要特性是,它提供了一个自然的保护措施,防止过拟合,因为它最大限度地减少了平均风险,同时考虑到模型复杂性引起的惩罚。值得注意的是,实证结果表明,存在一个最佳的信息率最小化的\emph{excess risk},这取决于可用的训练数据的性质和数量。层次文本分类的应用程序也进行了研究,扩展以前的作品。

英文摘要:

This paper investigates, from information theoretic grounds, a learning problem based on the principle that any regularity in a given dataset can be exploited to extract compact features from data, i.e., using fewer bits than needed to fully describe the data itself, in order to build meaningful representations of a relevant content (multiple labels). We begin by introducing the noisy lossy source coding paradigm with the log-loss fidelity criterion which provides the fundamental tradeoffs between the \emph{cross-entropy loss} (average risk) and the information rate of the features (model complexity). Our approach allows an information theoretic formulation of the \emph{multi-task learning} (MTL) problem which is a supervised learning framework in which the prediction models for several related tasks are learned jointly from common representations to achieve better generalization performance. Then, we present an iterative algorithm for computing the optimal tradeoffs and its global convergence is proven provided that some conditions hold. An important property of this algorithm is that it provides a natural safeguard against overfitting, because it minimizes the average risk taking into account a penalization induced by the model complexity. Remarkably, empirical results illustrate that there exists an optimal information rate minimizing the \emph{excess risk} which depends on the nature and the amount of available training data. An application to hierarchical text categorization is also investigated, extending previous works.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑