神经语言模型中语法性的线性表示

中文摘要 AI 辅助

神经语言模型是否具备基于语法性区分字符串的能力在计算语言学文献中仍是一个有争议的话题。现有证据大多依赖基于概率的度量。本文超越基于概率的评估，通过质量均值探测研究语法性是否编码在神经语言模型的内部表示中，测试语法和非语法句子在表示空间中是否系统分离，还考察了表示与相关属性的独立性及跨语法现象和语言的泛化性。结果表明语法性在多种预训练神经语言模型的句子表示中得到有力编码，这为语言模型句法知识本质的争论提供了新证据，也提供了不依赖字符串概率的语法能力评估框架。

英文摘要

Whether neural language models (NLMs) possess the ability to distinguish strings on the basis of their grammaticality remains a debated topic in the computational linguistics literature. Existing evidence has largely relied on probability-based measures, testing whether models assign higher probabilities to grammatical than ungrammatical strings. However, probability comparisons have been criticized as a measure for grammatical knowledge based on the assumption that grammaticality is inherently entangled with likelihood. Model-assigned probability is a function of many related sentence properties, such as lexical frequency, plausibility, and world knowledge. In this work, we move beyond probability-based evaluations and investigate whether grammaticality is encoded in the internal representations of NLMs. Using mass-mean probing, we test whether grammatical and ungrammatical sentences are systematically separated in representational space. We further examine the extent to which these representations are independent of sentence properties that are correlated with grammaticality, as well as their generalization across grammatical phenomena and languages. Our results provide evidence that grammaticality is robustly encoded in sentence representations of a wide range of pretrained NLMs, yielding clear representational separation on the dimension of grammaticality that cannot be fully explained by alternative sentence-level factors. Moreover, this encoding generalizes across a broad range of grammatical phenomena and to some degree, across languages, suggesting that grammaticality constitutes a coherent representational dimension in contemporary NLMs. These findings contribute new evidence to debates about the nature of syntactic knowledge in language models and offer a complementary framework for evaluating grammatical competence that is not dependent on string probabilities alone.

发表机构

• Johns Hopkins University（约翰·霍普金斯大学）
• Boston University（波士顿大学）

机构由 AI 辅助整理，请以论文原文为准。