arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2408.00386cs.LG
收藏

Transformer之后是什么?——连接深度学习思想的精选综述

What comes after transformers? -- A selective survey connecting ideas in deep learning

University of Liechtenstein(列支敦士登大学)

Johannes Schneider

AI总结:

本文综述了Transformer之后深度学习领域的新兴架构与优化思想,重点分析可能替代Transformer的创新方法及近期成功策略,并讨论了GPT、LLama和Gemini等前沿模型。

摘要:

自2017年以来,Transformer已成为人工智能领域事实上的标准模型,尽管其存在从能源效率低下到幻觉等诸多缺点。研究在改进Transformer的各个要素方面取得了大量进展,更广泛地说,深度学习领域涌现出许多关于架构、层、优化目标和优化技术的提案。研究人员很难在更宏观的层面上跟踪这些发展。我们为那些已经具备深度学习基础理解的读者提供了这些领域中许多重要的近期工作的全面概述。我们的关注点与其他工作不同,因为我们专门针对新颖的、可能具有颠覆性的Transformer替代方法以及近期深度学习的成功思想。我们希望这种对有影响力的近期工作和新颖思想的整体性、统一性处理能够帮助研究人员在深度学习的不同领域之间建立新的联系。我们识别并讨论了多种模式,这些模式总结了近十年来成功创新的关键策略,以及可以被视为冉冉升起的新星的工作。特别是,我们讨论了改进Transformer的尝试,涵盖(部分)经过验证的方法,如状态空间模型(state space models),同时也包括深度学习中看似有前景但尚未达到最先进结果的大胆想法。我们还讨论了近期最先进的模型,如OpenAI的GPT系列、Meta的LLama模型以及Google的Gemini模型家族。

英文摘要:

Transformers have become the de-facto standard model in artificial intelligence since 2017 despite numerous shortcomings ranging from energy inefficiency to hallucinations. Research has made a lot of progress in improving elements of transformers, and, more generally, deep learning manifesting in many proposals for architectures, layers, optimization objectives, and optimization techniques. For researchers it is difficult to keep track of such developments on a broader level. We provide a comprehensive overview of the many important, recent works in these areas to those who already have a basic understanding of deep learning. Our focus differs from other works, as we target specifically novel, alternative potentially disruptive approaches to transformers as well as successful ideas of recent deep learning. We hope that such a holistic and unified treatment of influential, recent works and novel ideas helps researchers to form new connections between diverse areas of deep learning. We identify and discuss multiple patterns that summarize the key strategies for successful innovations over the last decade as well as works that can be seen as rising stars. Especially, we discuss attempts on how to improve on transformers covering (partially) proven methods such as state space models but also including far-out ideas in deep learning that seem promising despite not achieving state-of-the-art results. We also cover a discussion on recent state-of-the-art models such as OpenAI's GPT series and Meta's LLama models and, Google's Gemini model family.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑