绘制混沌边缘：仅解码器Transformer模型可训练性中的分形边界
Mapping the Edge of Chaos: Fractal-Like Boundaries in The Trainability of Decoder-Only Transformer Models
AI总结：
本研究将微型神经网络训练结果边界的分形特征扩展到中等规模仅解码器Transformer，通过一致收敛度量分析注意力与全连接层学习率景观，发现可训练性前沿为多尺度自相似随机结构，稳定区域被混沌边界包围。
摘要：
在分形几何领域，复杂结构源于将参数空间划分为稳定与不稳定区域的简单迭代过程。同样，训练大型语言模型涉及迭代应用Adam等更新函数，即使轻微调整超参数也会使训练过程从收敛转向发散。微型神经网络的近期证据表明，分隔这些结果的边界具有分形特征。基于这些见解，本研究通过采用更一致的收敛度量，并检查注意力层和全连接层的学习率超参数景观，将其扩展到中等规模的仅解码器Transformer架构。结果显示，可训练性前沿并非简单阈值；相反，它在多尺度上形成自相似但看似随机的结构，具有统计一致且重复的模式。在此景观中，稳定收敛区域被复杂的混沌边界包围，体现了底层训练动态的敏感性。
英文摘要：
In the realm of fractal geometry, intricate structures emerge from simple iterative processes that partition parameter spaces into regions of stability and instability. Likewise, training large language models involves iteratively applying update functions, such as Adam, where even slight hyperparameter adjustments can shift the training process from convergence to divergence. Recent evidence from miniature neural networks suggests that the boundary separating these outcomes displays fractal characteristics. Building on these insights, this study extends them to medium-sized, decoder-only transformer architectures by employing a more consistent convergence measure and examining the learning rate hyperparameter landscape for attention and fully connected layers. The results show that the trainability frontier is not a simple threshold; rather, it forms a self-similar yet seemingly random structure at multiple scales, with statistically consistent and repeating patterns. Within this landscape, a region of stable convergence is surrounded by a complex chaotic border, illustrating the sensitive nature of the underlying training dynamics.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
