AI 中文总结
本研究通过合成语法实验揭示，神经语言模型在训练初期即形成类别级几何结构支持泛化，随后逐渐转向示例记忆，为理解模型学习机制提供了几何视角。
AI 中文摘要
我们研究了神经语言模型在学习轨迹中泛化与记忆的发展过程。通过使用受控的合成语法，我们考察了表征空间的几何结构以及模型在训练过程中的行为。我们发现，从学习的最早阶段开始，表征空间中未被观察到的标记占据的连续区域就被系统地结构化，形成支持对未验证组合进行泛化的类别级几何组织。因此，泛化从学习一开始就出现，而不是仅在大量记忆之后才出现。随着训练的延长，较大的模型越来越能区分已观察和未观察的语法组合。与此同时，支持类别级泛化的连续几何结构逐渐被破坏。这些结果表明，神经语言模型最初通过基于分类的泛化进行学习，随后逐渐转向更具体的示例记忆。
英文摘要
We investigate how generalization and memorization develop along the learning trajectory of neural language models. Using controlled synthetic grammars, we examine both the geometry of representation space and model behaviour over training. We find that continuous regions of representation space not occupied by observed tokens become systematically structured from the earliest stages of learning, forming category-level geometric organization that supports generalization to unattested combinations. Generalization therefore emerges from the beginning of learning rather than only after extensive memorization. With prolonged training, larger models increasingly distinguish observed from unobserved grammatical combinations. At the same time, the continuous geometric structure supporting category-level generalization is gradually destructed. Together, these results suggest that neural language models initially learn through categorization-based generalization, followed by a gradual transition toward more exemplar-specific memorization.
Comments5 figures in main text, 9 page main text
