arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2111.01635cs.LGcs.ITmath.ITstat.ML
收藏

基于Gibbs算法的迁移学习泛化误差的表征与理解

Characterizing and Understanding the Generalization Error of Transfer Learning with Gibbs Algorithm

Massachusetts Institute of Technology(麻省理工学院);University College London(伦敦大学学院)

Yuheng Bu, Gholamali Aminian, Laura Toni, Miguel Rodrigues, Gregory Wornell

摘要:

我们提供了一个基于吉布斯的迁移学习算法的泛化能力的信息论分析,专注于两个流行的迁移学习方法,$α$加权ERM和两阶段ERM。我们的主要结果是一个精确的泛化行为的特征,使用输出假设和目标训练样本之间的条件对称化KL信息给定的源样本。我们的结果也可以应用于提供新的分布自由的泛化误差上界这两个前面提到的吉布斯算法。我们的方法是多功能的,因为它也表征了这两个吉布斯算法的泛化误差和超额风险的渐近制度,在那里他们收敛到$α$加权的ERM和两阶段的ERM,分别。基于我们的理论结果,我们表明,迁移学习的好处可以被看作是一个偏差-方差权衡,与源分布引起的偏差和目标样本的缺乏引起的方差。我们相信这一观点可以指导实践中迁移学习算法的选择。

英文摘要:

We provide an information-theoretic analysis of the generalization ability of Gibbs-based transfer learning algorithms by focusing on two popular transfer learning approaches, $α$-weighted-ERM and two-stage-ERM. Our key result is an exact characterization of the generalization behaviour using the conditional symmetrized KL information between the output hypothesis and the target training samples given the source samples. Our results can also be applied to provide novel distribution-free generalization error upper bounds on these two aforementioned Gibbs algorithms. Our approach is versatile, as it also characterizes the generalization errors and excess risks of these two Gibbs algorithms in the asymptotic regime, where they converge to the $α$-weighted-ERM and two-stage-ERM, respectively. Based on our theoretical results, we show that the benefits of transfer learning can be viewed as a bias-variance trade-off, with the bias induced by the source distribution and the variance induced by the lack of target samples. We believe this viewpoint can guide the choice of transfer learning algorithms in practice.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑