arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2510.07591cs.CL
收藏

创建ConLangs以探测LLMs的元语言语法知识

Creating ConLangs to Probe the Metalinguistic Grammatical Knowledge of LLMs

University of Notre Dame(诺特达美大学);Sakana AI(萨卡纳人工智能);Notre Dame, IN, USA(印第安纳州诺特达美);Tokyo, Japan(日本东京)

Chihiro Taguchi, Richard Sproat

AI总结:

本文提出IASC系统,通过LLMs生成构造语言,探讨LLMs对语言和语言学概念的理解能力,实验显示不同LLM和语言规范在形态语法能力上有显著差异。

摘要:

我们提出一个系统,利用LLMs作为工具开发构造语言(ConLangs),称为IASC(交互代理系统for ConLangs)。该系统模块化,通过模块特定的提示生成语音学、形态学、句法、词汇、正写法和语法手册等组件。该方法具有代理特性,允许通过模块对先前步骤自动生成的评论进行细化。我们的主要目标是提供工具,使创建人工构造语言变得有趣和愉快。本文重点在于使用我们的ConLang框架作为新方法,探索LLMs对语言和语言学概念的理解程度,而非特定语言或百科知识。实验特别关注形态句法模块,显示不同LLM和语言规范在能力上有显著差异,系统处理更常见的语言类型比罕见类型更容易。所有代码均已发布。

英文摘要:

We present a system that uses LLMs as a tool in the development of Constructed Languages -- ConLangs, which we call IASC (Interactive Agentic System for ConLangs). The system is modular in that it creates each of the components -- phonology, morphology and syntax, lexicon, orthography, and grammatical handbook, using module-specific sets of prompts. The approach is agentic in that various modules allow for refining the output given automatically-generated commentary on a previous step. Our main goals are twofold. First, we aim to provide tools that facilitate an engaging and enjoyable experience in creating artificially constructed languages. Second, the focus of this paper is on using our ConLang framework as a novel way to explore what LLMs 'know' about language -- not what they know about any particular language or encyclopedic facts, but how much they know about and understand language and linguistic concepts. In the experiments, we particularly focus on the morphosyntax module and show that there is a fairly wide gulf in capabilities both among different LLMs and among different linguistic specifications, with it being notably easier for systems to deal with more typologically common patterns than rarer ones. All code is released.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑