arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2409.05925cs.DBcs.AIcs.CLcs.IR
收藏

评估大型语言模型的SPARQL能力

Assessing SPARQL capabilities of Large Language Models

Institute for Applied Informatics at Leipzig University(莱比锡大学应用信息学研究所);Leipzig University(莱比锡大学)

Lars-Peter Meyer, Johannes Frey, Felix Brei, Natanael Arndt

AI总结:

本文在LLM-KG-Bench框架中实现了多维度基准测试任务,定量评估了GPT、Gemini和Claude等模型处理SPARQL SELECT查询的开箱即用能力,发现生成语义正确的查询对LLMs仍具挑战性。

摘要:

将大型语言模型(LLMs)与知识图谱(KGs)集成,为知识驱动的应用提供了巨大的协同潜力。一种可能的集成方式是解释和生成形式语言,例如语义网中使用的语言,其中SPARQL是访问KGs的核心技术。本文聚焦于衡量LLMs开箱即用地处理SPARQL的能力,更具体地说,是采用定量方法评估LLMs处理SPARQL SELECT查询的能力。我们在LLM-KG-Bench框架中实现了各种基准测试任务,以对多个LLMs进行自动执行和评估。这些任务从语法、语义读取、语义创建以及知识图谱提示包含的角色等维度评估能力。借助这些新的基准测试任务,我们评估了部分GPT、Gemini和Claude模型。研究结果表明,处理SPARQL SELECT查询对LLMs而言仍具挑战性,且严重依赖于具体的LLM以及任务复杂度。尽管修复基本语法错误对评估中表现最佳的当前LLMs似乎不成问题,但在多种情况下,创建语义正确的SPARQL SELECT查询仍很困难。

英文摘要:

The integration of Large Language Models (LLMs) with Knowledge Graphs (KGs) offers significant synergistic potential for knowledge-driven applications. One possible integration is the interpretation and generation of formal languages, such as those used in the Semantic Web, with SPARQL being a core technology for accessing KGs. In this paper, we focus on measuring out-of-the box capabilities of LLMs to work with SPARQL and more specifically with SPARQL SELECT queries applying a quantitative approach. We implemented various benchmarking tasks in the LLM-KG-Bench framework for automated execution and evaluation with several LLMs. The tasks assess capabilities along the dimensions of syntax, semantic read, semantic create, and the role of knowledge graph prompt inclusion. With this new benchmarking tasks, we evaluated a selection of GPT, Gemini, and Claude models. Our findings indicate that working with SPARQL SELECT queries is still challenging for LLMs and heavily depends on the specific LLM as well as the complexity of the task. While fixing basic syntax errors seems to pose no problems for the best of the current LLMs evaluated, creating semantically correct SPARQL SELECT queries is difficult in several cases.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑