arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2510.19797cs.LG
收藏

Transformer 是线性分类任务中近乎最优的元学习器

Transformers are almost optimal metalearners for linear classification

Weizmann Institute of Science(魏兹曼科学研究院)

Roey Magen, Gal Vardi

AI总结:

该研究首次从理论证明经梯度下降训练的简化Transformer在线性分类元学习场景中近乎最优,仅需少量上下文示例即可泛化新任务,性能匹配已知共享子空间的最优学习器,且训练需求与环境维度无关。

摘要:

Transformer 已展现出令人瞩目的上下文学习(ICL)能力,这引发了一个问题:它们能否作为元学习器,仅利用少量上下文示例就能适配新任务,而无需任何额外训练。尽管近期的理论工作已研究了 Transformer 执行 ICL 的能力,但这些分析大多未涉及正式的元学习设定——元学习的目标是比逐个解决每个任务更高效地求解一组相关任务。本文首次提供了理论分析,证明经梯度下降训练的简化 Transformer 架构可在线性分类场景中作为近乎最优的元学习器。我们考虑一类自然的任务族,其中每个任务对应一个类条件高斯混合模型,其均值向量位于 R^d 的一个共享 k 维子空间中。我们证明,在足够多的此类任务上训练后,Transformer 仅需 O(k / R^4) 个上下文示例就能泛化到新任务,其中 R 表示测试时的信号强度。这一性能(几乎)与确切知晓共享子空间的最优学习器性能相当,且显著优于任何仅能访问上下文数据的学习器——后者需要 Ω(d / R^4) 个示例才能实现泛化。重要的是,我们得出的实现该结果所需的训练任务数量及每个任务的示例数量的界,与环境维度 d 无关。

英文摘要:

Transformers have demonstrated impressive in-context learning (ICL) capabilities, raising the question of whether they can serve as metalearners that adapt to new tasks using only a small number of in-context examples, without any further training. While recent theoretical work has studied transformers' ability to perform ICL, most of these analyses do not address the formal metalearning setting, where the objective is to solve a collection of related tasks more efficiently than would be possible by solving each task individually. In this paper, we provide the first theoretical analysis showing that a simplified transformer architecture trained via gradient descent can act as a near-optimal metalearner in a linear classification setting. We consider a natural family of tasks where each task corresponds to a class-conditional Gaussian mixture model, with the mean vectors lying in a shared $k$-dimensional subspace of $R^d$. After training on a sufficient number of such tasks, we show that the transformer can generalize to a new task using only $O(k / R^4)$ in-context examples, where $R$ denotes the signal strength at test time. This performance (almost) matches that of an optimal learner that knows exactly the shared subspace and significantly outperforms any learner that only has access to the in-context data, which requires $Ω(d / R^4)$ examples to generalize. Importantly, our bounds on the number of training tasks and examples per task needed to achieve this result are independent of the ambient dimension $d$.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑