arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2202.08876stat.MLcs.LG
收藏

一种使用单调变分不等式训练神经网络的替代方法

An alternative approach to train neural networks using monotone variational inequality

Georgia Institute of Technology(佐治亚理工学院);Duke University(杜克大学)

Chen Xu, Xiuyuan Cheng, Yao Xie

AI总结:

本研究受广义线性模型参数估计相关工作启发,提出将神经网络训练转化为单调变分不等式求解的替代方法,具备高效收敛与特定场景理论保证,适用于多类网络训练及预训练模型微调,性能媲美或优于随机梯度下降法。

摘要:

我们提出了一种利用单调向量场训练神经网络的替代方法,该思路受Juditsky与Nemirovski的开创性工作[Juditsky & Nemirovsky, 2019]启发,该工作最初用于解决广义线性模型(GLM)的参数估计问题,通过将原始非凸问题转化为求解单调变分不等式(VI)的凸问题实现。我们的方法可得到计算高效的流程,收敛速度快,且在单层神经网络训练、预训练模型最后一层微调等特殊场景下具备理论保证。该方法可用于在冻结底层的情况下更高效地微调预训练模型,这是部署大语言模型(LLM)等众多机器学习模型的关键步骤。我们验证了其在全连接(FC)神经网络、图神经网络(GNN)和卷积神经网络(CNN)训练中的适用性,并且在合成与真实网络数据预测任务的多项性能指标上,该方法的表现与随机梯度下降方法相当或更优。

英文摘要:

We propose an alternative approach to neural network training using the monotone vector field, an idea inspired by the seminal work of Juditsky and Nemirovski [Juditsky & Nemirovsky, 2019] developed originally to solve parameter estimation problems for generalized linear models (GLM) by reducing the original non-convex problem to a convex problem of solving a monotone variational inequality (VI). Our approach leads to computationally efficient procedures that converge fast and offer guarantee in some special cases, such as training a single-layer neural network or fine-tuning the last layer of the pre-trained model. Our approach can be used for more efficient fine-tuning of a pre-trained model while freezing the bottom layers, an essential step for deploying many machine learning models such as large language models (LLM). We demonstrate its applicability in training fully-connected (FC) neural networks, graph neural networks (GNN), and convolutional neural networks (CNN) and show the competitive or better performance of our approach compared to stochastic gradient descent methods on both synthetic and real network data prediction tasks regarding various performance metrics.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑