隐藏的孔洞：语言模型的拓扑学方面
Hidden Holes: topological aspects of language models
AI总结：
本文提出“穿孔”拓扑复杂度度量，比较 GPT 与门控循环语言模型的表示流形，发现后者拓扑复杂度更高且其变化模式仅存在于自然语言中，揭示了 Transformer 与 LSTM 架构在拓扑结构上的显著差异。
摘要：
我们探索在原始文本数据上训练的自回归神经语言模型中产生的表示流形的拓扑结构。为了研究它们的性质，我们引入计算代数拓扑工具，并以此为基础提出一种拓扑复杂度度量，我们称之为“穿孔”（perforation）。利用这一度量，我们研究基于 GPT 的大规模语言模型在训练过程中跨深度和时间的拓扑结构演化。随后，我们将这些模型与门控循环模型进行比较，结果表明后者表现出更高的拓扑复杂度，并且具有一种独特的变化模式，该模式普遍存在于所有自然语言中，但在合成生成的数据中不存在。本文基于研究这些模型在受自然语言文本语料库中的句子条件作用时所产生的向量云形状，对它们导出的表示流形进行了详细分析。本文提出的方法在该领域具有新颖性，并且基于目标读者可能不熟悉的数学工具。为帮助理解，我们介绍了必要的最低限度理论，并在附录中提供了额外的可视化。本文的主要贡献是关于 Transformer 与基于 LSTM 的神经架构相比在拓扑结构方面的一个引人注目的观察。这表明有必要进一步研究这些神经网络的数学性质，以理解大型 Transformer 语言模型的运作方式。我们希望这项工作能激励 NLP 社区在这一方向上开展进一步探索。
英文摘要：
We explore the topology of representation manifolds arising in autoregressive neural language models trained on raw text data. In order to study their properties, we introduce tools from computational algebraic topology, which we use as a basis for a measure of topological complexity, that we call perforation. Using this measure, we study the evolution of topological structure in GPT based large language models across depth and time during training. We then compare these to gated recurrent models, and show that the latter exhibit more topological complexity, with a distinct pattern of changes common to all natural languages but absent from synthetically generated data. The paper presents a detailed analysis of the representation manifolds derived by these models based on studying the shapes of vector clouds induced by them as they are conditioned on sentences from corpora of natural language text. The methods developed in this paper are novel in the field and based on mathematical apparatus that might be unfamiliar to the target audience. To help with that we introduce the minimum necessary theory, and provide additional visualizations in the appendices. The main contribution of the paper is a striking observation about the topological structure of the transformer as compared to LSTM based neural architectures. It suggests that further research into mathematical properties of these neural networks is necessary to understand the operation of large transformer language models. We hope this work inspires further explorations in this direction within the NLP community.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
