注入结构提示：利用语言模型研究语言学习中的归纳偏置
Injecting structural hints: Using language models to study inductive biases in language learning
AI总结：
本文通过向transformer语言模型注入递归层级处理、非上下文无关交叉关系和Zipfian词汇分布三种归纳偏置，发现非上下文无关关系是最佳归纳偏置，为理解人类与机器的语言学习机制提供了新假设。
摘要：
人类和大型语言模型都能够在没有显式结构监督的情况下学习语言。是哪些归纳偏置使这种学习成为可能？我们通过利用transformer语言模型来回答这一基本认知问题：我们通过在形式化结构数据上进行预训练，将归纳偏置注入语言模型，然后评估这些带有偏置的学习者学习类型学上多样化自然语言的能力。我们的实验设置构建了一个用于检验人类语言学习中归纳偏置假设的测试平台。我们研究了向模型注入三种归纳偏置的效果：1）递归、层级化处理，2）无法由上下文无关文法建模的交叉token-token关系，以及3）Zipfian幂律词汇分布。我们表明，非上下文无关关系构成了最佳的归纳偏置。我们的研究利用transformer模型的能力来进行受控语言学习实验，这些实验无法在人类身上进行，并提出了关于促进人类和机器语言学习的结构的假设。
英文摘要：
Both humans and large language models are able to learn language without explicit structural supervision. What inductive biases make this learning possible? We address this fundamental cognitive question by leveraging transformer language models: we inject inductive bias into language models by pretraining on formally-structured data, and then evaluate the biased learners' ability to learn typologically-diverse natural languages. Our experimental setup creates a testbed for hypotheses about inductive bias in human language learning. We investigate the effect of injecting models with three types of inductive bias: 1) recursive, hierarchical processing, 2) crossing token-token relationships that can't be modeled by context-free grammars, and 3) a Zipfian power-law vocabulary distribution. We show that non-context-free relationships form the best inductive biases. Our study leverages the capabilities of transformer models to run controlled language learning experiments that are not possible to run on humans, and surfaces hypotheses about the structures that facilitate language learning in both humans and machines.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
