基于数据采样和多任务优化的DNN训练框架
A Novel DNN Training Framework via Data Sampling and Multi-Task Optimization
摘要:
传统的DNN训练范例通常依赖于一个训练集和一个验证集,这是通过以某种方式划分用于训练的注释数据集(即总训练集)而获得的。训练集用于训练模型,而验证集用于在训练进行时估计训练模型的泛化性能,以避免过拟合。在这一范式中存在两个主要问题。首先,由于与测试数据的潜在不匹配,验证集可能很难保证泛化性能的无偏估计。其次,训练DNN对应于解决复杂的优化问题,这很容易陷入局部最优,从而导致不期望的训练结果。为了解决这些问题,我们提出了一种新的DNN训练框架。它通过随机分割从总训练集生成多对训练集和验证集,在每对上训练预先指定结构的DNN模型,同时使有用的知识(例如,有希望的网络参数),以经由多任务优化转移到其他模型训练过程,并且输出所有训练模型中的最佳模型,其在来自所有对的验证集上具有总体最佳性能。该框架中的知识转移机制不仅可以帮助模型训练过程摆脱局部最优,从而提高训练效率,还可以通过对一个模型训练过程施加来自其他模型训练过程的隐式正则化来提高泛化性能。我们实现了所提出的框架,并行GPU集群上的实现,并将其应用到训练几个广泛使用的DNN模型。实验结果表明,所提出的框架优于传统的训练范式。
英文摘要:
Conventional DNN training paradigms typically rely on one training set and one validation set, obtained by partitioning an annotated dataset used for training, namely gross training set, in a certain way. The training set is used for training the model while the validation set is used to estimate the generalization performance of the trained model as the training proceeds to avoid over-fitting. There exist two major issues in this paradigm. Firstly, the validation set may hardly guarantee an unbiased estimate of generalization performance due to potential mismatching with test data. Secondly, training a DNN corresponds to solve a complex optimization problem, which is prone to getting trapped into inferior local optima and thus leads to undesired training results. To address these issues, we propose a novel DNN training framework. It generates multiple pairs of training and validation sets from the gross training set via random splitting, trains a DNN model of a pre-specified structure on each pair while making the useful knowledge (e.g., promising network parameters) obtained from one model training process to be transferred to other model training processes via multi-task optimization, and outputs the best, among all trained models, which has the overall best performance across the validation sets from all pairs. The knowledge transfer mechanism featured in this new framework can not only enhance training effectiveness by helping the model training process to escape from local optima but also improve on generalization performance via implicit regularization imposed on one model training process from other model training processes. We implement the proposed framework, parallelize the implementation on a GPU cluster, and apply it to train several widely used DNN models. Experimental results demonstrate the superiority of the proposed framework over the conventional training paradigm.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。