发表机构
Northwestern University; University of Southern California; Yale University; California Institute of Technology（西北大学; 南加州大学; 耶鲁大学; 加州理工学院）
机构由 AI 辅助整理，请以论文原文为准。AI 中文总结
本研究通过拆分结构探针的评估指标UASL，发现影响其变异的两个关键因素，揭示了大语言模型句法表征的抽象程度及对嵌入空间几何属性的依赖性。
AI 中文摘要
休伊特（Hewitt）和曼宁（Manning）提出结构探针，用于从神经语言模型的潜在表征中重构句法树，其评估方式为在带注释语料库上正确重构的句法树边比例（以无向未标注依附分数衡量）。本文对该指标进行拆分，按标签计算无向依附分数（UASL），分别评估各句法关系的重构准确率，发现不同关系间存在与语言学区分重叠的重要差异。此外，确定了两个可预测UASL在各关系间大部分变异的因素：（i）相关词语间线性距离（对数尺度）的均值与离散度；（ii）句法关系中心的多样性（相似度感知熵）。这些结果在一系列模型规模与架构中均成立，揭示了语言模型中句法表征的抽象程度，以及该表征对嵌入空间几何属性的依赖性。
英文摘要
Structural probes were introduced by Hewitt and Manning to reconstruct syntactic trees from a neural language model's latent representations. They are evaluated by calculating the proportion of syntactic tree edges correctly reconstructed over an annotated corpus (as measured by undirected unlabeled attachment score). Here, we disaggregate this measure, considering undirected attachment score by label (UASL), which assesses the reconstruction accuracy of each syntactic relation separately, establishing important differences among relations that overlap linguistic distinctions. Moreover, we identify two factors that predict most of UASL's variability across relations: (i) the mean and dispersion of the linear distance (on a log scale) between the related words, and (ii) the diversity (similarity-aware entropy) of the syntactic relation's head. These results, which hold across a range of model sizes and architectures, shed light on the degree of abstraction of the representation of syntax in language models and the dependence of such representation on geometric properties of the embedding space.
Comments29 pages (9 main text, 18 appendix), 20 figures, 7 tables. Code and data: https://github.com/jpvigneaux/structural-probes-labelwise-analysis
