你的上下文不是数组：揭示Transformer中的随机访问局限性
Your Context Is Not an Array: Unveiling Random Access Limitations in Transformers
AI总结：
本研究针对Transformer大语言模型的长度泛化失效问题，通过奇偶校验任务分析发现其根源在于模型无法在上下文窗口内执行随机内存访问，并通过相关方法验证及注意力图可视化佐证了该假设。
摘要：
尽管基于Transformer的大语言模型近期取得了诸多成功，却展现出了令人意外的失效模式，这类失效模式的一个典型例子是其无法实现长度泛化——即在推理阶段解决比训练期间见过的实例更长的问题。在本研究中，我们通过对简单的奇偶校验任务下的模型行为开展详细分析，进一步探究了这一失效的根本原因。分析表明，长度泛化失效与模型无法在其上下文窗口内执行随机内存访问密切相关。我们通过验证两类方法的有效性为该假设提供了支撑证据：一类是规避索引需求的方法，另一类是通过基于内容的寻址间接实现随机token访问的方法。我们还通过注意力图可视化，进一步展示了随机内存访问失效的出现位置与作用机制。
英文摘要：
Despite their recent successes, Transformer-based large language models show surprising failure modes. A well-known example of such failure modes is their inability to length-generalize: solving problem instances at inference time that are longer than those seen during training. In this work, we further explore the root cause of this failure by performing a detailed analysis of model behaviors on the simple parity task. Our analysis suggests that length generalization failures are intricately related to a model's inability to perform random memory accesses within its context window. We present supporting evidence for this hypothesis by demonstrating the effectiveness of methodologies that circumvent the need for indexing or that enable random token access indirectly, through content-based addressing. We further show where and how the failure to perform random memory access manifests through attention map visualizations.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
