从弗雷格到ChatGPT：语言、认知与深度神经网络中的组合性
From Frege to chatGPT: Compositionality in language, cognition, and deep neural networks
AI总结：
本文面向哲学、认知科学等领域读者，综述机器学习领域关于组合性的实证研究，介绍赋予神经网络组合泛化能力的两种方法，指出LLM预训练可视为元学习并带来该能力，探讨其对人类认知研究的启示与未来方向。
摘要：
组合性长期以来被认为是人类智能背后的一项关键解释性属性：任意概念都可以组合成全新的复杂组合，使得人类能够从有限的学习经验中获得开放的、潜在无限的表达能力。颇具影响力的论点曾认为，神经网络无法解释行为的这一方面，导致许多人将其排除在人类认知的可行模型之外。然而在过去十年间，与其前身共享相同基础设计原则的现代深度神经网络（DNNs）已主导人工智能领域，展现出机器有史以来最先进的认知行为。特别是大型语言模型（LLMs）——即在大规模文本语料库上训练以预测下一个词的深度神经网络——已被证明能够完成复杂行为，例如写出语法无误的复杂句法句子、生成有说服力的推理链，甚至编写原创计算机程序——所有这些行为都曾被认为需要组合性处理。在本章节中，我们面向哲学、认知科学和神经科学领域的广泛读者，综述了机器学习领域的最新实证研究，将近期突破置于关于组合性的哲学论证的更广阔背景下。具体而言，我们的综述重点介绍了赋予神经网络组合泛化能力的两种方法：（1）架构归纳偏置，以及（2）元学习（即学会学习）。我们还展示了相关研究发现，表明LLM预训练可以被理解为一种元学习，因此能够以类似的方式为深度神经网络配备组合泛化能力。最后，我们讨论了这些发现可能对人类认知中的组合性研究产生的启示，并提出了未来的研究方向。
英文摘要：
Compositionality has long been considered a key explanatory property underlying human intelligence: arbitrary concepts can be composed into novel complex combinations, permitting the acquisition of an open ended, potentially infinite expressive capacity from finite learning experiences. Influential arguments have held that neural networks fail to explain this aspect of behavior, leading many to dismiss them as viable models of human cognition. Over the last decade, however, modern deep neural networks (DNNs), which share the same fundamental design principles as their predecessors, have come to dominate artificial intelligence, exhibiting the most advanced cognitive behaviors ever demonstrated in machines. In particular, large language models (LLMs), DNNs trained to predict the next word on a large corpus of text, have proven capable of sophisticated behaviors such as writing syntactically complex sentences without grammatical errors, producing cogent chains of reasoning, and even writing original computer programs -- all behaviors thought to require compositional processing. In this chapter, we survey recent empirical work from machine learning for a broad audience in philosophy, cognitive science, and neuroscience, situating recent breakthroughs within the broader context of philosophical arguments about compositionality. In particular, our review emphasizes two approaches to endowing neural networks with compositional generalization capabilities: (1) architectural inductive biases, and (2) metalearning, or learning to learn. We also present findings suggesting that LLM pretraining can be understood as a kind of metalearning, and can thereby equip DNNs with compositional generalization abilities in a similar way. We conclude by discussing the implications that these findings may have for the study of compositionality in human cognition and by suggesting avenues for future research.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
