AI 中文总结
本文研究语言模型语义空间中关系几何的表征情况，通过三类语言模型在6种语义关系上的实验，发现非对称关系的表征更清晰，且不同模型依赖的信息来源存在差异。
AI 中文摘要
在生成单词的向量表示方面，当前语言模型已取得高质量成果，但尚不清楚语义关系知识在多大程度上体现在此类模型构建的语义空间几何结构中。为回答该问题，本文从三个视角研究这类语义空间的关系几何：首先，考察与目标词存在特定关系的词（称为关系项）是否占据语义空间中的同一区域，以及不同关系对应的区域是否彼此区分；其次，验证语义空间在多大程度上反映关系的已知属性，如对称性、非对称性和传递性；最后，探究目标词与关系项的哪些信息对关系几何更重要：是词形还是上下文。本文在6种语义关系上对因果语言模型、掩码语言模型和扩散语言模型开展实验，结果显示：非对称关系的关系项相对清晰地占据语义空间的不同区域；非对称关系的属性在语义空间中的编码程度仅为中等，但优于对称关系；在评估的模型中，词法信息对因果语言模型更重要，而上下文信息对掩码语言模型和扩散语言模型更重要。上述结果实证表明，语义空间中不同关系的几何表征程度并不一致，这意味着仅从分布信息中学习语义关系的效果存在差异。
英文摘要
When it comes to generating vector representations of words, current language models are achieving high-quality results. However, what is not known is the extent to which knowledge about semantic relations is represented in the geometry of the semantic spaces created in this way. In order to answer this question, we study the relation geometry of such semantic spaces from three perspectives. We first examine whether words standing in a particular relation to a target word~(called relata) occupy the same region in semantic space, and whether the regions corresponding to different relations are distinct from each other. We then verify to what extent semantic spaces reflect certain well-known properties of relations, such as symmetry, asymmetry, and transitivity. Finally, we consider which information about the target words and relata is more important for relation geometry: their surface forms, or their contexts. We conduct experiments on six semantic relations using causal, masked, and diffusion language models. The results show that relata in asymmetric relations relatively clearly occupy a distinct region in semantic space. Asymmetric relations' properties are only moderately well encoded in the semantic space, yet better than those of symmetric ones. Furthermore, when considering the question which information source has the strongest impact on results amongst the models we evaluated, we find that lexical information tends to be more important for the causal language model, whereas contextual information is more important for the masked and diffusion language models. Our results empirically show that relation geometry is not equally well-represented for all relations in semantic space, suggesting that there is a difference in how well semantic relations might be learned from distributional information alone.
CommentsManuscript under review
