arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 1710.03282cs.LG
收藏

检查点集成:从单个训练过程中集成方法

Checkpoint Ensembles: Ensemble Methods from a Single Training Process

University of Washington(华盛顿大学)

Hugh Chen, Scott Lundberg, Su-In Lee

摘要:

我们提出了检查点集成方法,可以在单个训练过程中学习集成模型。虽然检查点集成可以应用于任何参数迭代学习技术,但在这里我们关注的是神经网络。神经网络的可组合和简单的神经元使得它能够捕捉许多个体和特征之间的交互作用。然而,小样本大小和采样噪声可能导致训练数据中的模式不代表特征和结果之间的真实关系。作为一种解决方案,经常使用训练期间的正则化(例如dropout)。然而,正则化并不是万灵药--它并不能完美地解决过拟合问题。即使使用dropout这样的方法,在实践中也通常使用两种方法。首先是利用独立于训练集的验证集来决定何时停止训练。其次是使用集成方法来进一步减少过拟合并利用局部最优值(即对几个模型的预测进行平均)。在本文中,我们将探索检查点集成-一种将这两种思想结合在一个训练过程中的简单技术。检查点集成通过在单个训练过程中平均来自最佳模型的“检查点”的预测来提高性能。我们使用三个真实世界的数据集-文本,图像和电子健康记录数据-使用三种预测模型:香草神经网络,卷积神经网络和长短期记忆网络,以表明检查点集成优于现有的方法:一种方法是通过最小验证分数选择模型,两种方法是通过权重平均模型。我们的研究结果还表明,检查点集成捕获传统集成提供的性能增益的一部分。

英文摘要:

We present the checkpoint ensembles method that can learn ensemble models on a single training process. Although checkpoint ensembles can be applied to any parametric iterative learning technique, here we focus on neural networks. Neural networks' composable and simple neurons make it possible to capture many individual and interaction effects among features. However, small sample sizes and sampling noise may result in patterns in the training data that are not representative of the true relationship between the features and the outcome. As a solution, regularization during training is often used (e.g. dropout). However, regularization is no panacea -- it does not perfectly address overfitting. Even with methods like dropout, two methodologies are commonly used in practice. First is to utilize a validation set independent to the training set as a way to decide when to stop training. Second is to use ensemble methods to further reduce overfitting and take advantage of local optima (i.e. averaging over the predictions of several models). In this paper, we explore checkpoint ensembles -- a simple technique that combines these two ideas in one training process. Checkpoint ensembles improve performance by averaging the predictions from "checkpoints" of the best models within single training process. We use three real-world data sets -- text, image, and electronic health record data -- using three prediction models: a vanilla neural network, a convolutional neural network, and a long short term memory network to show that checkpoint ensembles outperform existing methods: a method that selects a model by minimum validation score, and two methods that average models by weights. Our results also show that checkpoint ensembles capture a portion of the performance gains that traditional ensembles provide.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑