arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2605.15413cs.LG
收藏

Transformer可扩展性危机:现代语言模型中性能瓶颈的首次全面实证分析

Transformer Scalability Crisis: The First Comprehensive Empirical Analysis of Performance Walls in Modern Language Models

Research Division, BrightMind AI(BrightMind AI研究部);Texas Tech University(德克萨斯科技大学);University of Texas at Arlington(德克萨斯大学阿灵顿分校)

Mahdi Naser Moghadasi, Faezeh Ghaderi

AI总结:

本文通过评估118种transformer模型,揭示了模型在序列长度扩展时的性能瓶颈,发现随着序列长度增加,模型处理能力显著下降,挑战了传统扩展假设。

摘要:

尽管transformer架构在自然语言处理中取得了显著成功,但其可扩展性限制仍缺乏系统性的实证分析。本文首次对118种transformer模型进行了大规模评估,涵盖七种不同的架构类别,揭示了根本性的性能瓶颈,表现为硬性部署限制。我们的系统性基准测试方法揭示了关键的可扩展性危机:虽然88.1%的模型能够处理最多512个token的序列,但在1024个token时降至44.9%,在2048个token时完全失败。通过严格分析加载时间、内存消耗和计算效率,我们证明压缩模型在参数效率(649.2 tokens/sec/M参数)上优于大型生成模型(12.5 tokens/sec/M)。我们的发现挑战了主流的扩展假设,并提供了首次定量证据,表明理论上的O(n²)注意力复杂度转化为可测量的性能瓶颈。本工作建立了新的transformer评估基准方法,并为生产环境中的实际部署决策提供了关键见解。

英文摘要:

Despite the remarkable success of transformer architectures in natural language processing, their scalability limitations remain poorly understood through systematic empirical analysis. This paper presents the first comprehensive large-scale evaluation of 118 transformer models across seven distinct architectural categories, revealing fundamental performance walls that manifest as hard deployment constraints. Our systematic benchmarking methodology uncovers a critical scalability crisis: while 88.1% of models successfully process sequences up to 512 tokens, this drops dramatically to 44.9% at 1024 tokens, with complete failure (0%) at 2048 tokens. Through rigorous analysis of loading times, memory consumption, and computational efficiency across sequence lengths from 128 to 2048 tokens, we demonstrate that compressed models achieve superior parameter efficiency (649.2 tokens/sec/M parameters) compared to large generative models (12.5 tokens/sec/M). Our findings challenge prevailing scaling assumptions and provide the first quantitative evidence that the theoretical O(n2) attention complexity translates into measurable performance walls. This work establishes new benchmarking methodologies for transformer evaluation and provides critical insights for practical deployment decisions in production environments.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑