arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2303.16504cs.LGstat.ML
收藏

过参数指数回归

An Over-parameterized Exponential Regression

The University of Washington(华盛顿大学);Adobe Research(奥多比研究院)

Yeqi Gao, Sridhar Mahadevan, Zhao Song

摘要:

在过去的几年里,有大量的研究集中在研究ReLU激活函数上,目的是通过过度参数化来实现神经网络收敛。然而,大型语言模型(LLM)领域的最新发展引发了人们对指数激活函数的兴趣,特别是在注意力机制中。在数学上,我们使用指数激活函数定义神经函数$F: \mathbb{R}^{d \times m} \times \mathbb{R}^d \rightarrow \mathbb{R}$。给定一组带有标签$\{(x_1, y_1), (x_2, y_2), \dots, (x_n, y_n)\} \subset \mathbb{R}^d \times \mathbb{R}$的数据点,其中$n$表示数据的数量。这里,$F(W(t),x)$可以表示为$F(W(t),x) := \sum_{r=1}^m a_r \exp(\langle w_r, x \rangle)$,其中$m$表示神经元的数量,$w_r(t)$是时间$t$处的权重。在文献中,$a_r$是固定的权重,并且在训练过程中从未改变。我们用随机高斯分布初始化权重$W(0) \in \mathbb{R}^{d \times m}$,使得$w_r(0) \sim \mathcal{N}(0, I_d)$和从每个$r \in [m]$的随机符号分布初始化$a_r$。使用梯度下降算法,我们可以找到一个权重$W(T)$,使得$\| F(W(T), X) - y \|_2 \leq ε$以概率$1-δ$成立,其中$ε\in (0,0.1)$和$m = Ω(n^{2+o(1)}\log(n/δ))$。为了优化过度参数化边界$m$,我们采用了先前研究中的几种紧密分析技术[Song和Yang arXiv 2019,Munteanu,Omlor,Song和Woodruff ICML 2022]。

英文摘要:

Over the past few years, there has been a significant amount of research focused on studying the ReLU activation function, with the aim of achieving neural network convergence through over-parametrization. However, recent developments in the field of Large Language Models (LLMs) have sparked interest in the use of exponential activation functions, specifically in the attention mechanism. Mathematically, we define the neural function $F: \mathbb{R}^{d \times m} \times \mathbb{R}^d \rightarrow \mathbb{R}$ using an exponential activation function. Given a set of data points with labels $\{(x_1, y_1), (x_2, y_2), \dots, (x_n, y_n)\} \subset \mathbb{R}^d \times \mathbb{R}$ where $n$ denotes the number of the data. Here $F(W(t),x)$ can be expressed as $F(W(t),x) := \sum_{r=1}^m a_r \exp(\langle w_r, x \rangle)$, where $m$ represents the number of neurons, and $w_r(t)$ are weights at time $t$. It's standard in literature that $a_r$ are the fixed weights and it's never changed during the training. We initialize the weights $W(0) \in \mathbb{R}^{d \times m}$ with random Gaussian distributions, such that $w_r(0) \sim \mathcal{N}(0, I_d)$ and initialize $a_r$ from random sign distribution for each $r \in [m]$. Using the gradient descent algorithm, we can find a weight $W(T)$ such that $\| F(W(T), X) - y \|_2 \leq ε$ holds with probability $1-δ$, where $ε\in (0,0.1)$ and $m = Ω(n^{2+o(1)}\log(n/δ))$. To optimize the over-parameterization bound $m$, we employ several tight analysis techniques from previous studies [Song and Yang arXiv 2019, Munteanu, Omlor, Song and Woodruff ICML 2022].

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑