AI 中文总结
本研究首次系统探究大语言模型的反向关系方向性，用含5457个实例的基准评估5种开源模型，发现其反向关系分类存在不对称性，关系描述与实体表示会影响性能。
AI 中文摘要
大语言模型（LLMs）在文本到知识图谱生成及相关任务中已取得优异性能，但目前仍不清楚它们是否准确建模了具有方向依赖性的反向关系语义，这类关系中论元顺序反转会改变关系含义（例如“母亲”与“孩子”）。据我们所知，本研究首次对大语言模型中的反向关系方向性开展系统研究，使用包含5457个实例、涵盖27种不同反向关系标签的基准数据集，在多项选择提示框架下评估5种开源大语言模型，并通过将原始实体替换为合成实体和掩码实体，进一步考察关系描述和实体表示的影响。研究结果表明，大语言模型在反向关系分类中存在系统性不对称性，关系描述并非总能提升性能，且模型性能对实体表示的变化较为敏感。
英文摘要
Large language models (LLMs) have achieved strong performance on text-to-knowledge graph generation and related tasks. Nevertheless, it is still unclear whether they accurately model the direction-dependent semantics of inverse relations, in which reversing the order of the arguments alters the meaning of a relation (e.g., \textit{mother} versus \textit{child}). To the best of our knowledge, this work presents the first systematic study of inverse relation directionality in LLMs, using a benchmark consisting of 5,457 instances spanning 27 distinct inverse relation labels. We evaluate five open-source LLMs under a multiple-choice prompting framework and further examine the influence of relation descriptions and entity representations by substituting the original entities with synthetic and masked entities. Our findings reveal systematic asymmetries in inverse relation classification across LLMs, indicate that relation descriptions do not consistently improve performance, and show that model performance can be sensitive to variations in entity representations.
CommentsThe preprint is under review in a venue
