小型Transformer中的计数:注意力与前馈层之间的微妙相互作用
Counting in Small Transformers: The Delicate Interplay between Attention and Feed-Forward Layers
AI总结:
本研究分析简单Transformer块处理直方图计数任务的解决方案,识别基于关系和库存的两种计数策略,揭示注意力与前馈层功能分布,证明微小设计变化对基础聚合选择任务的显著影响。
摘要:
除了规模考量外,架构设计选择深刻塑造了Transformer的解空间。本研究分析了简单Transformer块在处理直方图任务(即计数序列中的项目)时实现的解决方案。尽管任务简单,但它揭示了预测性能、词汇量与嵌入大小、令牌混合机制以及前馈层容量之间的复杂相互作用。我们确定了Transformer采用的两种理论计数策略:基于关系的计数和基于库存的计数,每种策略都为该任务定义了不同的学习机制。这些策略决定了功能在注意力层和前馈层之间的分布方式。我们进一步表明,当嵌入维度相对较小时,添加softmax和序列起始令牌(beginning-of-sequence tokens)可提高鲁棒性。对训练后模型的实证分析密切证实了各种架构的学习机制以及训练过程中这些策略的形成。我们证明,仅需聚合和选择的基础任务会因微小的设计变化而受到显著影响。
英文摘要:
Next to scaling considerations, architectural design choices profoundly shape the solution space of transformers. In this work, we analyze the solutions simple transformer blocks implement when tackling the histogram task: counting items in sequences. Despite its simplicity, this task reveals a complex interplay between predictive performance, vocabulary and embedding sizes, token-mixing mechanisms, and feed-forward layer capacity. We identify two theoretical counting strategies transformers adopt, relation-based and inventory-based counting, each defining distinct learning regimes for the task. These strategies dictate how functionality is distributed between attention and feed-forward layers. We further show that adding softmax and beginning-of-sequence tokens allow for more robustness when embedding dimensions are comparatively small. Empirical introspection of trained models closely confirms both the learning regimes of the various architectures and the formation of these strategies during training. We demonstrate how a basic task that requires only aggregation and selection is significantly impacted by minor design changes.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。