高斯匹配与复制:一种研究Transformer归纳的极简基准
Gaussian Match-and-Copy: A Minimalist Benchmark for Studying Transformer Induction
AI总结:
本文提出高斯匹配与复制基准,用于研究Transformer的匹配与复制机制,通过隔离长距离检索信号,分析模型在检索能力上的差异。
摘要:
匹配与复制是大型语言模型在推理过程中用于从上下文中检索匹配标记然后复制其后继的核心检索基本操作。然而,理解这种行为如何在自然数据上出现具有挑战性,因为检索和记忆是交织在一起的。为了分离这两个方面,我们引入了高斯匹配与复制(GMC),一种极简的基准,通过纯二阶相关信号来隔离长距离检索。数值研究显示,此任务保留了变压器在实践中发展匹配与复制电路的关键定性方面,并通过检索能力区分架构。我们还分析了简化注意力设置中的优化动态。尽管许多解决方案在回归目标下可能是先验可能的,包括那些不实现检索的方案,但我们识别出一个隐含偏置区域,在此区域中梯度下降驱动参数发散,而其方向与最大边际分隔符一致,从而产生硬匹配选择。我们证明了在达到消失经验损失的GD轨迹下,此最大边际对齐的数学证明在显式技术条件下成立。
英文摘要:
Match-and-copy is a core retrieval primitive used at inference time by large language models to retrieve a matching token from the context then copy its successor. Yet, understanding how this behavior emerges on natural data is challenging because retrieval and memorization are entangled. To disentangle the two, we introduce Gaussian Match-and-Copy (GMC), a minimalist benchmark that isolates long-range retrieval through pure second-order correlation signals. Numerical investigations show that this task retains key qualitative aspects of how Transformers develop match-and-copy circuits in practice, and separates architectures by their retrieval capabilities. We also analyze the optimization dynamics in a simplified attention setting. Although many solutions are a priori possible under a regression objective, including ones that do not implement retrieval, we identify an implicit-bias regime in which gradient descent drives the parameters to diverge while their direction aligns with the max-margin separator, yielding hard match selection. We prove this max-margin alignment for GD trajectories that reach vanishing empirical loss under explicit technical conditions.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。