大型语言模型（LLMs）已为人类社会带来了重大且深刻的变革。这些模型在自然语言理解与生成方面展现出卓越的能力，在多个领域推动了各种进步并产生了广泛影响。本文在两种注意力相关回归的形式化框架下研究上下文学习。给定矩阵 $A_1 \in \mathbb{R}^{n \times d}$、$A_2 \in \mathbb{R}^{n \times d}$ 和 $B \in \mathbb{R}^{n \times n}$，目标是求解特定的优化问题：归一化版本 $\min_{X} \\| D(X)^{-1} \exp(A_1 X A_2^\top) - B \\|_F^2$ 和重缩放版本 $\\| \exp(A_1 X A_2^\top) - D(X) \cdot B \\|_F^2$。其中 $D(X) := \mathrm{diag}( \exp(A_1 X A_2^\top) {\bf 1}_n )$。我们的回归问题与先前关于softmax相关回归的研究具有相似之处。此前的研究已广泛探讨了与softmax回归相关的回归技术：归一化版本 $\\| \langle \exp(Ax) , {\bf 1}_n \rangle^{-1} \exp(Ax) - b \\|_2^2$ 和重缩放版本 $\\| \exp(Ax) - \langle \exp(Ax), {\bf 1}_n \rangle b \\|_2^2$。与以往方法不同，我们采用向量化技术来处理矩阵形式下的回归问题。该方法将维度从 $d$ 扩展到 $d^2$，类似于前述回归问题的形式化表达。在完成回归函数的Lipschitz分析之后，我们得出了关于上下文学习的主要结果。
英文摘要：
Large language models (LLMs) have brought significant and transformative changes in human society. These models have demonstrated remarkable capabilities in natural language understanding and generation, leading to various advancements and impacts across several domains.
We consider the in-context learning under two formulation for attention related regression in this work. Given matrices $A_1 \in \mathbb{R}^{n \times d}$, and $A_2 \in \mathbb{R}^{n \times d}$ and $B \in \mathbb{R}^{n \times n}$, the purpose is to solve some certain optimization problems: Normalized version $\min_{X} \| D(X)^{-1} \exp(A_1 X A_2^\top) - B \|_F^2$ and Rescaled version $\| \exp(A_1 X A_2^\top) - D(X) \cdot B \|_F^2$. Here $D(X) := \mathrm{diag}( \exp(A_1 X A_2^\top) {\bf 1}_n )$.
Our regression problem shares similarities with previous studies on softmax-related regression. Prior research has extensively investigated regression techniques related to softmax regression: Normalized version $\| \langle \exp(Ax) , {\bf 1}_n \rangle^{-1} \exp(Ax) - b \|_2^2$ and Resscaled version $\| \exp(Ax) - \langle \exp(Ax), {\bf 1}_n \rangle b \|_2^2 $
In contrast to previous approaches, we adopt a vectorization technique to address the regression problem in matrix formulation. This approach expands the dimension from $d$ to $d^2$, resembling the formulation of the regression problem mentioned earlier.
Upon completing the lipschitz analysis of our regression function, we have derived our main result concerning in-context learning.
ETH Zürich（苏黎世联邦理工学院）；University of Toronto（多伦多大学）；Google Research, Brain Team（谷歌研究院大脑团队）；DeepMind（深度思维公司）；Institute for Advanced Study（高等研究院）
The Gumbel-Max trick is the basis of many relaxed gradient estimators. These estimators are easy to implement and low variance, but the goal of scaling them comprehensively to large combinatorial distributions is still outstanding. Working within the perturbation model framework, we introduce stochastic softmax tricks, which generalize the Gumbel-Softmax trick to combinatorial spaces. Our framework is a unified perspective on existing relaxed estimators for perturbation models, and it contains many novel relaxations. We design structured relaxations for subset selection, spanning trees, arbore…
Despite great popularity of applying softmax to map the non-normalised outputs of a neural network to a probability distribution over predicting classes, this normalised exponential transformation still seems to be artificial. A theoretic framework that incorporates softmax as an intrinsic component is still lacking. In this paper, we view neural networks embedding softmax from an information-theoretic perspective. Under this view, we can naturally and mathematically derive log-softmax as an inherent component in a neural network for evaluating the conditional mutual information between networ…
University of California, Los Angeles（加州大学洛杉矶分校）；University of British Columbia（不列颠哥伦比亚大学）；Emory University（埃默里大学）
We present ADMM-Softmax, an alternating direction method of multipliers (ADMM) for solving multinomial logistic regression (MLR) problems. Our method is geared toward supervised classification tasks with many examples and features. It decouples the nonlinear optimization problem in MLR into three steps that can be solved efficiently. In particular, each iteration of ADMM-Softmax consists of a linear least-squares problem, a set of independent small-scale smooth, convex problems, and a trivial dual variable update. Solution of the least-squares problem can be be accelerated by pre-computing a f…
Borealis AI（Borealis人工智能公司）；Vector Institute（向量研究所）；University of Guelph（圭尔夫大学）
ML models often operate within the context of a larger system that can adapt its response when the ML model is uncertain, such as falling back on safe defaults or a human in the loop. This commonly encountered operational context calls for principled techniques for training ML models with the option to abstain from predicting when uncertain. Selective neural networks are trained with an integrated option to abstain, allowing them to learn to recognize and optimize for the subset of the data distribution for which confident predictions can be made. However, optimizing selective networks is chal…
Statistical Advantage of Softmax Attention: Insights from Single-Location Regression
Institute of Mathematics, EPFL（EPFL数学研究所）；Statistical Physics of Computation Laboratory, École Polytechnique Fédérale de Lausanne（计算统计物理实验室，瑞士联邦理工学院）；Inria, École Normale Supérieure, PSL Research University（Inria，巴黎高等师范学院，PSL研究大学）；Laboratoire de Mathématiques d’Orsay, Université Paris Saclay and Institut Universitaire de France（奥赛数学实验室，巴黎萨克雷大学及法国国家科学院）；Departement d’Informatique, École Normale Supérieure, PSL & CNRS（信息科学系，巴黎高等师范学院，PSL与CNRS）
Instituto Superior Técnico & Instituto de Telecomunicações, Universidade de Lisboa, Portugal（里斯本大学理工学院及电信研究所，葡萄牙）；Champalimaud Research, Lisbon, Portugal（Champalimaud研究，葡萄牙里斯本）；TransPerfect, Lisbon, Portugal（TransPerfect，葡萄牙里斯本）；Carnegie Mellon University, Pittsburgh, USA（卡内基梅隆大学，美国匹兹堡）
This paper proposes a method for solving multivariate regression and classification problems using piecewise linear predictors over a polyhedral partition of the feature space. The resulting algorithm that we call PARC (Piecewise Affine Regression and Classification) alternates between (i) solving ridge regression problems for numeric targets, softmax regression problems for categorical targets, and either softmax regression or cluster centroid computation for piecewise linear separation, and (ii) assigning the training points to different clusters on the basis of a criterion that balances pre…
University of Amsterdam（阿姆斯特丹大学）；Inception Institute of Artificial Intelligence（起源人工智能研究院）
Multi-task learning aims to explore task relatedness to improve individual tasks, which is of particular significance in the challenging scenario that only limited data is available for each task. To tackle this challenge, we propose variational multi-task learning (VMTL), a general probabilistic inference framework for learning multiple related tasks. We cast multi-task learning as a variational Bayesian inference problem, in which task relatedness is explored in a unified manner by specifying priors. To incorporate shared knowledge into each task, we design the prior of a task to be a learna…
Prominently used in support vector machines and logistic regressions, kernel functions (kernels) can implicitly map data points into high dimensional spaces and make it easier to learn complex decision boundaries. In this work, by replacing the inner product function in the softmax layer, we explore the use of kernels for contextual word classification. In order to compare the individual kernels, experiments are conducted on standard language modeling and machine translation tasks. We observe a wide range of performances across different kernel settings. Extending the results, we look at the g…
Function approximation is a powerful approach for structuring large decision problems that has facilitated great achievements in the areas of reinforcement learning and game playing. Regression counterfactual regret minimization (RCFR) is a simple algorithm for approximately solving imperfect information games with normalized rectified linear unit (ReLU) parameterized policies. In contrast, the more conventional softmax parameterization is standard in the field of reinforcement learning and yields a regret bound with a better dependence on the number of actions. We derive approximation error-a…
Multinomial logistic regression, also known by other names such as multiclass logistic regression and softmax regression, is a fundamental classification method that generalizes binary logistic regression to multiclass problems. A recently work proposed a faster gradient called $\texttt{quadratic gradient}$ that can accelerate the binary logistic regression training, and presented an enhanced Nesterov's accelerated gradient (NAG) method for binary logistic regression. In this paper, we extend this work to multiclass logistic regression and propose an enhanced Adaptive Gradient Algorithm (Adagr…
We consider a discriminative learning (regression) problem, whereby the regression function is a convex combination of k linear classifiers. Existing approaches are based on the EM algorithm, or similar techniques, without provable guarantees. We develop a simple method based on spectral techniques and a `mirroring' trick, that discovers the subspace spanned by the classifiers' parameter vectors. Under a probabilistic assumption on the feature vector distribution, we prove that this approach has nearly optimal statistical efficiency.
In-Context Algorithm Emulation in Fixed-Weight Transformers
Center for Foundation Models and Generative AI, Northwestern University（西北大学基础模型与生成式人工智能中心）；Department of Computer Science, Northwestern University（西北大学计算机科学系）；Ensemble AI；Engineering Science, University of Toronto（多伦多大学工程科学系）；Department of Statistics and Data Science, Northwestern University（西北大学统计与数据科学系）
University of Cambridge（剑桥大学）；Google Research（谷歌研究院）；Columbia University（哥伦比亚大学）；The Alan Turing Institute（艾伦·图灵研究所）
The problem of efficient approximation of a linear operator induced by the Gaussian or softmax kernel is often addressed using random features (RFs) which yield an unbiased approximation of the operator's result. Such operators emerge in important applications ranging from kernel methods to efficient Transformers. We propose parameterized, positive, non-trigonometric RFs which approximate Gaussian and softmax-kernels. In contrast to traditional RF approximations, parameters of these new methods can be optimized to reduce the variance of the approximation, and the optimum can be expressed in cl…
Pontifícia Univesidade Católica do Rio de Janeiro（里约热内卢天主教大学）
The goal of Ordinal Regression is to find a rule that ranks items from a given set. Several learning algorithms to solve this prediction problem build an ensemble of binary classifiers. Ranking by Projecting uses interdependent binary perceptrons. These perceptrons share the same direction vector, but use different bias values. Similar approaches use independent direction vectors and biases. To combine the binary predictions, most of them adopt a simple counting heuristics. Here, we introduce a novel cumulative sum scoring function to combine the binary predictions. The proposed score value ag…
Tsinghua-Berkeley Shenzhen Institute（清华-伯克利深圳学院）；Massachusetts Institute of Technology（麻省理工学院）
We consider the problem of identifying universal low-dimensional features from high-dimensional data for inference tasks in settings involving learning. For such problems, we introduce natural notions of universality and we show a local equivalence among them. Our analysis is naturally expressed via information geometry, and represents a conceptually and computationally useful analysis. The development reveals the complementary roles of the singular value decomposition, Hirschfeld-Gebelein-Rényi maximal correlation, the canonical correlation and principle component analyses of Hotelling and Pe…
Center for Computational and Data-Intensive Science and Engineering（计算与数据密集型科学与工程中心）；Skolkovo Institute of Science and Technology（斯科尔科沃科学技术研究所）
We present a different view on stochastic optimization, which goes back to the splitting schemes for approximate solutions of ODE. In this work, we provide a connection between stochastic gradient descent approach and first-order splitting scheme for ODE. We consider the special case of splitting, which is inspired by machine learning applications and derive a new upper bound on the global splitting error for it. We present, that the Kaczmarz method is the limit case of the splitting scheme for the unit batch SGD for linear least squares problem. We support our findings with systematic empiric…
