基于Transformer的谱方法学习
Learning Spectral Methods by Transformers
AI总结:
本研究从理论和实验两方面证明,多层预训练Transformer可通过学习谱方法完成无监督统计估计任务,为理解其学习机制提供新视角。
摘要:
Transformer作为现代大语言模型(LLM)的基础构件展现出显著优势。本研究探讨Transformer在执行无监督学习任务时的能力。我们证明,给定足够多的预训练样本,多层Transformer能够自行学习算法,并针对新样本执行统计估计任务。这种学习范式有别于上下文学习设置,与人类大脑通过过往经验习得技能的学习过程相似。理论层面,我们证明预训练Transformer能够学习谱方法,并以双类高斯混合模型的分类任务为例展开验证。我们的证明采用算法设计技术,具有构造性。研究结果建立在多层Transformer架构与实际应用中迭代恢复算法的相似性之上。实验层面,我们通过在合成数据集与真实世界数据集上执行主成分分析(PCA)和聚类任务,验证了多层(预训练)Transformer在无监督学习上的强大能力。
英文摘要:
Transformers demonstrate significant advantages as the building block of modern LLMs. In this work, we study the capacities of Transformers in performing unsupervised learning. We show that multi-layered Transformers, given a sufficiently large set of pre-training instances, are able to learn the algorithms themselves and perform statistical estimation tasks given new instances. This learning paradigm is distinct from the in-context learning setup and is similar to the learning procedure of human brains where skills are learned through past experience. Theoretically, we prove that pre-trained Transformers can learn the spectral methods and use the classification of bi-class Gaussian mixture model as an example. Our proof is constructive using algorithmic design techniques. Our results are built upon the similarities of multi-layered Transformer architecture with the iterative recovery algorithms used in practice. Empirically, we verify the strong capacity of the multi-layered (pre-trained) Transformer on unsupervised learning through the lens of both the PCA and the Clustering tasks performed on the synthetic and real-world datasets.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。