arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2106.02793cs.LGcs.AI
收藏

通过遍历权空间测地线求解混合机器学习任务

Solving hybrid machine learning tasks by traversing weight space geodesics

Caltech(加州理工学院)

Guruprasad Raghavan, Matt Thomson

摘要:

机器学习问题具有内在几何结构,因为包括神经网络的权重空间的中心对象和与特定任务相关联的损失函数可以被视为对给定机器学习问题的内在几何结构进行编码。因此,几何概念可以应用于分析和理解机器学习策略的理论特性以及开发新的算法。在本文中,我们通过基于微分几何的统一框架来解决机器学习中三个看似无关的开放问题。具体来说,我们将神经网络的权重空间视为具有黎曼度量的流形,该度量对特定任务的性能进行编码。通过定义一个度量,我们可以在权重空间中构建测地线,最小长度路径,这些路径代表了在特定任务上具有等效或接近等效功能性能的网络集。然后,我们遍历测地线路径,同时确定满足第二个目标的网络。受几何洞察力的启发,我们将测地线框架应用于3个主要应用:(i)网络稀疏化(ii)通过构建一系列目标上的高性能网络来减轻灾难性遗忘,以及(iii)在非凸损失景观中找到连接深度网络的不同局部最优值的高精度路径。我们的结果是在MNIST,CIFAR-10/100上训练的各种网络架构(MLP,VGG 11/16)上获得的。广义上,我们介绍了一个几何框架,统一了一系列的机器学习目标,并可以应用于多个类别的神经网络架构。

英文摘要:

Machine learning problems have an intrinsic geometric structure as central objects including a neural network's weight space and the loss function associated with a particular task can be viewed as encoding the intrinsic geometry of a given machine learning problem. Therefore, geometric concepts can be applied to analyze and understand theoretical properties of machine learning strategies as well as to develop new algorithms. In this paper, we address three seemingly unrelated open questions in machine learning by viewing them through a unified framework grounded in differential geometry. Specifically, we view the weight space of a neural network as a manifold endowed with a Riemannian metric that encodes performance on specific tasks. By defining a metric, we can construct geodesic, minimum length, paths in weight space that represent sets of networks of equivalent or near equivalent functional performance on a specific task. We, then, traverse geodesic paths while identifying networks that satisfy a second objective. Inspired by the geometric insight, we apply our geodesic framework to 3 major applications: (i) Network sparsification (ii) Mitigating catastrophic forgetting by constructing networks with high performance on a series of objectives and (iii) Finding high-accuracy paths connecting distinct local optima of deep networks in the non-convex loss landscape. Our results are obtained on a wide range of network architectures (MLP, VGG11/16) trained on MNIST, CIFAR-10/100. Broadly, we introduce a geometric framework that unifies a range of machine learning objectives and that can be applied to multiple classes of neural network architectures.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑