arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2311.04524cs.DBcs.AI
收藏

通过RDF知识图谱和句子相似性验证ChatGPT事实

Validating ChatGPT Facts through RDF Knowledge Graphs and Sentence Similarity

FORTH(希腊研究与技术基金会);ICS(信息学与电信研究所);University of Crete(克里特大学)

Michalis Mountantonakis, Yannis Tzitzikas

AI总结:

本文提出一种利用RDF知识图谱和句子相似性验证ChatGPT事实的流水线,通过DBpedia和LODsyndesis等图谱及嵌入算法,在2,000个事实基准上验证了85.3%的正确事实并纠正了58%的错误事实。

摘要:

由于ChatGPT提供的详细回答缺乏依据,甚至对于知名人物、事件和地点也可能出现错误事实,本文提出了一种新颖的流水线,将ChatGPT的回答以RDF形式检索,并尝试使用一个或多个RDF知识图谱(KG)来验证ChatGPT事实。为此,我们利用DBpedia和LODsyndesis(一个聚合知识图谱,包含来自400个不同领域RDF知识图谱的20亿条三元组)以及短句嵌入,并引入一种算法,该算法返回最相关的三元组及其来源和置信度分数。这实现了对ChatGPT回答的验证,并为其补充依据和来源。为了评估该服务(以及此类服务),我们创建了一个评估基准,包含2,000个ChatGPT事实;具体而言,1,000个关于著名希腊人物的事实,500个关于希腊热门地点的事实,以及500个与希腊相关的事件事实。这些事实经过人工标注(约73%的ChatGPT事实是正确的,27%的事实是错误的)。结果令人鼓舞;以整个基准为例,我们成功验证了ChatGPT正确事实中的85.3%,并为58%的错误ChatGPT事实找到了正确答案。

英文摘要:

Since ChatGPT offers detailed responses without justifications, and erroneous facts even for popular persons, events and places, in this paper we present a novel pipeline that retrieves the response of ChatGPT in RDF and tries to validate the ChatGPT facts using one or more RDF Knowledge Graphs (KGs). To this end we leverage DBpedia and LODsyndesis (an aggregated Knowledge Graph that contains 2 billion triples from 400 RDF KGs of many domains) and short sentence embeddings, and introduce an algorithm that returns the more relevant triple(s) accompanied by their provenance and a confidence score. This enables the validation of ChatGPT responses and their enrichment with justifications and provenance. To evaluate this service (such services in general), we create an evaluation benchmark that includes 2,000 ChatGPT facts; specifically 1,000 facts for famous Greek Persons, 500 facts for popular Greek Places, and 500 facts for Events related to Greece. The facts were manually labelled (approximately 73% of ChatGPT facts were correct and 27% of facts were erroneous). The results are promising; indicatively for the whole benchmark, we managed to verify the 85.3% of the correct facts of ChatGPT and to find the correct answer for the 58% of the erroneous ChatGPT facts.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑