Over the past few years, there has been a significant amount of research focused on studying the ReLU activation function, with the aim of achieving neural network convergence through over-parametrization. However, recent developments in the field of Large Language Models (LLMs) have sparked interest in the use of exponential activation functions, specifically in the attention mechanism.
Mathematically, we define the neural function $F: \mathbb{R}^{d \times m} \times \mathbb{R}^d \rightarrow \mathbb{R}$ using an exponential activation function. Given a set of data points with labels $\{(x_1, y_1), (x_2, y_2), \dots, (x_n, y_n)\} \subset \mathbb{R}^d \times \mathbb{R}$ where $n$ denotes the number of the data. Here $F(W(t),x)$ can be expressed as $F(W(t),x) := \sum_{r=1}^m a_r \exp(\langle w_r, x \rangle)$, where $m$ represents the number of neurons, and $w_r(t)$ are weights at time $t$. It's standard in literature that $a_r$ are the fixed weights and it's never changed during the training. We initialize the weights $W(0) \in \mathbb{R}^{d \times m}$ with random Gaussian distributions, such that $w_r(0) \sim \mathcal{N}(0, I_d)$ and initialize $a_r$ from random sign distribution for each $r \in [m]$.
Using the gradient descent algorithm, we can find a weight $W(T)$ such that $\| F(W(T), X) - y \|_2 \leq ε$ holds with probability $1-δ$, where $ε\in (0,0.1)$ and $m = Ω(n^{2+o(1)}\log(n/δ))$. To optimize the over-parameterization bound $m$, we employ several tight analysis techniques from previous studies [Song and Yang arXiv 2019, Munteanu, Omlor, Song and Woodruff ICML 2022].
MMGP: a Mesh Morphing Gaussian Process-based machine learning method for regression of physical problems under non-parameterized geometrical variability
Over-parameterized Adversarial Training: An Analysis Overcoming the Curse of Dimensionality
Princeton University(普林斯顿大学);Institute for Advanced Study(高等研究院)
对抗性训练是一种流行的方法,使神经网络对对抗性扰动具有鲁棒性。在实践中,对抗训练导致低鲁棒训练损失。然而,对于为什么在自然条件下会发生这种情况,仍然缺乏严格的解释。最近,标准(非对抗性)监督训练的收敛理论由{ \em very overparametrized}网的各种团体开发。目前还不清楚如何将这些结果扩展到对抗训练,因为最小-最大目标。最近,Gao等人使用在线学习的工具朝着这个方向迈出了第一步,但是他们要求网络的宽度在输入维度$d$上为\emph{exponential},并且具有不自然的激活函数。我们的工作证明了在自然假设和ReLU激活下,对于\emph{polynomial}宽度而不是指数宽度,收敛到低鲁棒训练损失。我们证明的关键要素是表明初始化附近的ReLU网络可以近似阶跃函数,这可能是独立的兴趣。
Hierarchical Mixtures-of-Experts for Exponential Family Regression Models with Generalized Linear Mean Functions: A Survey of Approximation and Consistency Results
Sharp Generalization for Nonparametric Regression in Interpolation Space by Over-Parameterized Neural Networks Trained with Preconditioned Gradient Descent and Early Stopping
School of Computing and Augmented Intelligence(计算与增强智能学院);Arizona State University(亚利桑那州立大学);VecML Inc.(VecML公司)
Resampled Confidence Regions with Exponential Shrinkage for the Regression Function of Binary Classification
Institute for Computer Science and Control (SZTAKI), Hungarian Research Network (HUN-REN)(计算机科学与控制研究所(SZTAKI)、匈牙利研究网络(HUN-REN));Dept. of Probability Theory and Statistics, Institute of Mathematics, Eötvös Loránd University (ELTE)(概率论与统计系,数学研究所,埃格德·洛朗德大学(ELTE))
Gaussian Process regression over discrete probability measures: on the non-stationarity relation between Euclidean and Wasserstein Squared Exponential Kernels
University of Milano-Bicocca(米兰比可卡大学);OAKS srl(OAKS有限责任公司)
Efficient Gradient Estimation for Parameterized Quantum Systems with Lie Algebraic Symmetries
Department of Computer Sciences, Indiana University, Bloomington, IN, USA(印第安纳大学计算机科学系,印第安纳州布卢明顿);Department of Computer Sciences, Purdue University, West Lafayette, IN, USA(普渡大学计算机科学系,印第安纳州西拉法叶)
Exponential Convergence of (Stochastic) Gradient Descent for Separable Logistic Regression
Undergraduate Programme Indian Institute of Science, Bangalore(印度科学研究院布达佩斯分校本科项目);Dept. of Computer Science and Automation Indian Institute of Science, Bangalore(计算机科学与自动化系印度科学研究院布达佩斯分校);Inria, Ecole Normale Supérieure PSL Research University, Paris, France(法国巴黎PSL研究大学埃cole Normale Supérieure与Inria)
Efficient improper learning for online logistic regression
INRIA - Département d’Informatique de l’École Normale Supérieure(法国国家信息与自动化研究所 - 巴黎高等师范学院信息学系);PSL Research University(巴黎文理研究大学)
我们考虑在线逻辑回归的设置,并考虑半径为B的2球的后悔。已知(参见[Hazan et al.,2014]),任何适当的算法,具有对数遗憾的样本数(表示为n)必然遭受指数乘法常数在B。在这项工作中,我们设计了一个有效的不当的算法,避免了这个指数常数,同时保持对数遗憾。的确,[福斯特等人,2018]表明,下界不适用于不适当的算法,并提出了一种策略,基于指数权重与禁止计算复杂性。我们的新算法基于正则化的经验风险最小化与代理损失满足遗憾缩放为O(Blog(Bn)),每轮时间复杂度为O(d^2)。
The Parameterized Complexity of Computing the VC-Dimension
Université Clermont Auvergne, CNRS, Mines Saint-Étienne, Clermont Auvergne INP, LIMOS(克莱蒙特-奥弗涅大学、国家科学研究中心、圣艾蒂安矿业学院、克莱蒙特-奥弗涅理工学院、LIMOS实验室);Telefónica Scientific Research Barcelona, Spain(西班牙巴塞罗那电信科学研究所);Indian Institute of Science Education and Research Pune Pune, India(印度班加罗尔印度科学教育与研究学院)