arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2407.00958cs.AIcs.CLcs.LG
收藏

动态通用逼近理论:基于Transformer的大语言模型的基础理论

Dynamic Universal Approximation Theory: The Basic Theory for Transformer-based Large Language Models

The Hong Kong Polytechnic University(香港理工大学)

Wei Wang, Qing Li

AI总结:

该研究针对Transformer类大语言模型的理论基础缺失问题,引入通用逼近理论(UAT)作为理论框架,旨在解释Transformer的有效性、ICL能力、LoRA微调增强及模型剪枝实用性等核心问题。

摘要:

语言模型已成为人工智能领域的核心研究方向,尤其是随着ChatGPT等突破性创新的出现。大规模Transformer网络迅速成为推动自然语言处理算法发展的主流方法。这些模型基于Transformer架构构建,能够实现高度模拟人类交流的交互,且凭借海量知识储备,甚至可辅助指导人类完成任务。尽管其能力出众且复杂度不断提升,一个关键问题仍悬而未决——大语言模型(LLM)的理论基础。是什么让Transformer在支撑翻译、编码等智能语言应用时如此高效?大语言模型的上下文学习(In-Context Learning, ICL)能力的底层机制是什么?LoRA方案是如何增强大语言模型微调效果的?又是什么支撑了大语言模型剪枝的实用性?为解答这些关键问题并探究大语言模型内部的技术策略,我们利用通用逼近理论(Universal Approximation Theory, UAT)提供理论背景,阐明支撑这些技术进展的内在机制。

英文摘要:

Language models have emerged as a critical area of focus in artificial intelligence, particularly with the introduction of groundbreaking innovations like ChatGPT. Large-scale Transformer networks have quickly become the leading approach for advancing natural language processing algorithms. Built on the Transformer architecture, these models enable interactions that closely mimic human communication and, equipped with extensive knowledge, can even assist in guiding human tasks. Despite their impressive capabilities and growing complexity, a key question remains-the theoretical foundations of large language models (LLMs). What makes Transformer so effective for powering intelligent language applications, such as translation and coding? What underlies LLMs' ability for In-Context Learning (ICL)? How does the LoRA scheme enhance the fine-tuning of LLMs? And what supports the practicality of pruning LLMs? To address these critical questions and explore the technological strategies within LLMs, we leverage the Universal Approximation Theory (UAT) to offer a theoretical backdrop, shedding light on the mechanisms that underpin these advancements.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑