分析语言模型嵌入空间中语言关系的线性度
Analysing the Linearity of Linguistic Relations in Language Model Embedding Spaces
查看机构详情
- ADAPT Research Centre（ADAPT研究中心）
- Trinity College Dublin（都柏林圣三一大学）
- Indian Institute of Technology Bombay（印度理工学院孟买分校）
机构由 AI 辅助整理，请以论文原文为准。
浏览论文内容
中文总结 AI 辅助
本文提出框架分析语言模型嵌入中语言关系的线性编码，发现屈折和派生关系近完美线性，词典和百科关系误差大，且RoBERTa和ModernBERT优于GloVe。
中文摘要 AI 辅助
我们提出了一个框架，用于分析不同语言关系在语言模型嵌入空间中被线性编码的强度。我们通过约束线性近似对相关和不相关的词对进行形式化定义，并将其应用于扩展的BATS数据集，该数据集涵盖GloVe、RoBERTa和ModernBERT中的屈折、派生、词典和百科关系。实验表明，屈折和派生关系具有近乎完美的线性编码，但词典和百科关系的误差显著更高，尤其对于一对多和多对多关联。我们还发现，RoBERTa和ModernBERT通常比GloVe更线性地编码关系。这些结果表明，我们的框架可以揭示哪些关系结构在嵌入中最具线性可及性，为跨模型探测和比较关系几何提供了一种紧凑工具。
英文摘要
We propose a framework to analyse how strongly different linguistic relations are linearly encoded in language model embedding spaces. We formalise linear encoding via a constrained linear approximation over related and unrelated word pairs and apply this to an extended BATS dataset covering inflectional, derivational, lexicographic, and encyclopedic relations in GloVe, RoBERTa, and ModernBERT. Our experiments show near-perfect linear encodings for inflectional and derivational relations, but substantially higher errors for lexicographic and encyclopedic relations, especially for one-to-many and many-to-many associations. We also find that RoBERTa and ModernBERT generally encode relations more linearly than GloVe. These results indicate that our framework can reveal which relational structures are most linearly accessible in embeddings, offering a compact tool for probing and comparing relational geometry across models.
