arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2409.12293cs.LGcs.NAmath.NAstat.ML
收藏

线性系统的上下文学习:泛化理论及其在算子学习中的应用

In-Context Learning of Linear Systems: Generalization Theory and Applications to Operator Learning

School of Mathematics, University of Minnesota(明尼苏达大学数学学院);Department of Mathematics and Statistics, University of Massachusetts Amherst(马萨诸塞大学阿默斯特分校数学与统计学系)

Frank Cole, Yulong Lu, Wuzhe Xu, Tianhao Zhang

AI总结:

本文研究线性Transformer在上下文中求解线性系统的泛化理论,提出任务多样性作为域外泛化的充要条件,并应用于PDE算子学习,数值实验验证了理论。

摘要:

我们研究了使用线性Transformer架构在上下文中求解线性系统的理论保证。对于域内泛化,我们提供了神经缩放定律,该定律以训练和推理中使用的任务数量及样本大小来界定泛化误差。对于域外泛化,我们发现训练期间所见任务的分布对训练后的Transformer在任务分布偏移下的行为至关重要。我们引入了一种新颖的任务多样性概念,并证明它定义了预训练Transformer在任务分布偏移下泛化的充分必要条件。我们还探索了上下文学习线性系统的应用,例如用于偏微分方程的上下文算子学习。最后,我们提供了一些数值实验来验证所建立的理论。

英文摘要:

We study theoretical guarantees for solving linear systems in-context using a linear transformer architecture. For in-domain generalization, we provide neural scaling laws that bound the generalization error in terms of the number of tasks and sizes of samples used in training and inference. For out-of-domain generalization, we find that the behavior of trained transformers under task distribution shifts depends crucially on the distribution of the tasks seen during training. We introduce a novel notion of task diversity and show that it defines a necessary and sufficient condition for pre-trained transformers generalize under task distribution shifts. We also explore applications of learning linear systems in-context, such as to in-context operator learning for PDEs. Finally, we provide some numerical experiments to validate the established theory.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑