面向涌现语言的语素归纳

Emergent Linguistic Rules from Inducing Decision Trees: Disambiguating Discourse Clue Words

We apply decision tree induction to the problem of discourse clue word sense disambiguation with a genetic algorithm. The automatic partitioning of the training set which is intrinsic to decision tree induction gives rise to linguistically viable rules.

A Theory of Emergent In-Context Learning as Implicit Structure Induction

Scaling large language models (LLMs) leads to an emergent capacity to learn in-context from example demonstrations. Despite progress, theoretical understanding of this phenomenon remains limited. We argue that in-context learning relies on recombination of compositional operations found in natural language data. We derive an information-theoretic bound showing how in-context learning abilities arise from generic next-token prediction when the pretraining distribution has sufficient amounts of compositional structure, under linguistically motivated assumptions. A second bound provides a theoret…

大型语言模型中涌现的符号机制支持抽象推理

Emergent Symbolic Mechanisms Support Abstract Reasoning in Large Language Models

通过分析LLM内部机制，发现其抽象推理依赖由符号抽象、符号归纳和检索三个计算阶段构成的涌现符号架构，为符号与神经网络之争提供了新证据。

The Grammar of Emergent Languages

In this paper, we consider the syntactic properties of languages emerged in referential games, using unsupervised grammar induction (UGI) techniques originally designed to analyse natural language. We show that the considered UGI techniques are appropriate to analyse emergent languages and we then study if the languages that emerge in a typical referential game setup exhibit syntactic structure, and to what extent this depends on the maximum message length and number of symbols that the agents are allowed to use. Our experiments demonstrate that a certain message length and vocabulary size are…

作为所罗门诺夫归纳可计算近似的大语言模型

Large Language Models as Computable Approximations to Solomonoff Induction

该研究首次建立大语言模型与算法信息论的形式化关联，证明其训练与下一词元预测近似所罗门诺夫归纳，为多种涌现现象提供统一解释，并提出低置信度样本优先的少样本选择策略，经实验验证可显著提升性能，尤其适用于小模型。

大型语言模型中的涌现类比推理

Emergent Analogical Reasoning in Large Language Models

本研究对比了GPT-3与人类在类比推理任务上的表现，发现GPT-3展现出惊人的抽象模式归纳能力，在多数场景下与人类相当甚至超越，表明大型语言模型已获得零样本解决类比问题的涌现能力。