arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2302.04369cs.LG
收藏

基于最大均值离散的深度神经网络学习的无监督学习

Unsupervised Learning of Initialization in Deep Neural Networks via Maximum Mean Discrepancy

New York University(纽约大学);Prescient Design, Genentech(基因泰克Prescient Design部门)

Cheolhyoung Lee, Kyunghyun Cho

摘要:

尽管随机梯度下降最近在深度学习中取得了成功,但通常很难用不适当的初始参数来训练深度神经网络。即使训练成功,已知初始参数配置也可能对泛化产生负面影响。在本文中,我们提出了一个无监督的算法,找到良好的初始化输入数据,给定的下游任务是d-路分类。我们首先注意到,参数空间中的每个参数配置对应于d路分类的一个特定下游任务。然后,我们推测,学习的成功直接关系到下游任务在初始参数附近的多样性。因此,我们设计了一个算法,鼓励小扰动的初始参数配置导致一组不同的d-路分类任务。换句话说,所提出的算法确保任何下游任务的解决方案接近初始参数配置。我们经验评估所提出的算法在各种任务来自MNIST与一个完全连接的网络。在这些实验中,我们观察到我们的算法提高了大多数任务的平均测试准确度,并且当标记的示例数量较少时,这种提高更大。

英文摘要:

Despite the recent success of stochastic gradient descent in deep learning, it is often difficult to train a deep neural network with an inappropriate choice of its initial parameters. Even if training is successful, it has been known that the initial parameter configuration may negatively impact generalization. In this paper, we propose an unsupervised algorithm to find good initialization for input data, given that a downstream task is d-way classification. We first notice that each parameter configuration in the parameter space corresponds to one particular downstream task of d-way classification. We then conjecture that the success of learning is directly related to how diverse downstream tasks are in the vicinity of the initial parameters. We thus design an algorithm that encourages small perturbation to the initial parameter configuration leads to a diverse set of d-way classification tasks. In other words, the proposed algorithm ensures a solution to any downstream task to be near the initial parameter configuration. We empirically evaluate the proposed algorithm on various tasks derived from MNIST with a fully connected network. In these experiments, we observe that our algorithm improves average test accuracy across most of these tasks, and that such improvement is greater when the number of labelled examples is small.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑