语言模型究竟在建模什么？Transformer、自动机与思维格式
What do language models model? Transformers, automata, and the format of thought
AI总结：
本文为大型语言模型是对语料库的建模提供非贬低性辩护，基于Transformer的线性计算格式与人类超线性能力对比，并借捷径自动机提出正面解释，强调语言作为话语机器的意义。
摘要：
大型语言模型究竟在建模什么？它们是否揭示了人类能力的某些方面，抑或仅仅是我们用于训练它们的语料库的模型？我对后一种立场给出一种非贬低性的辩护。认知科学告诉我们，人类的语言能力依赖于超线性的计算格式。相比之下，Transformer架构至多支持线性格式的处理。这一论证将主要依赖于Transformer计算架构的某些不变性。随后，我提出一个关于Transformer在做什么的正面解释，重点关注Liu等人（2022）关于捷径自动机的引人入胜的推测。最后，我阐述为何我认为这并非一个过于贬低性的故事。语言不仅仅是表达内在状态的手段，也是一种‘话语机器’，它让我们在适当的语境下生成新的语言。我们已学会以某种方式使用这一技术；大型语言模型也学会了使用它，但却是通过非常不同的途径。
英文摘要：
What do large language models actually model? Do they tell us something about human capacities, or are they models of the corpus we've trained them on? I give a non-deflationary defence of the latter position. Cognitive science tells us that linguistic capabilities in humans rely supralinear formats for computation. The transformer architecture, by contrast, supports at best a linear formats for processing. This argument will rely primarily on certain invariants of the computational architecture of transformers. I then suggest a positive story about what transformers are doing, focusing on Liu et al. (2022)'s intriguing speculations about shortcut automata. I conclude with why I don't think this is a terribly deflationary story. Language is not (just) a means for expressing inner state but also a kind of 'discourse machine' that lets us make new language given appropriate context. We have learned to use this technology in one way; LLMs have also learned to use it too, but via very different means.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
