arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2304.07772cs.CLcs.LG
收藏

从自然语言问题生成神经SPARQL查询的综合评估

A Comprehensive Evaluation of Neural SPARQL Query Generation from Natural Language Questions

Polytechnique Montreal(蒙特利尔综合理工学院);LAMA-WeST Lab(LAMA-WeST实验室)

Papa Abdou Karim Karou Diallo, Samuel Reyd, Amal Zouaq

AI总结:

本文系统评估了基于神经机器翻译的SPARQL查询生成方法,证明复制机制和数据标注策略能显著提升预训练及非预训练语言模型的URI生成准确性,而大型语言模型表现未达预期。

摘要:

近年来,面向SPARQL查询生成的神经机器翻译(NMT)领域取得了显著发展。将复制机制(copy mechanism)与传统编码器-解码器架构相结合,以及使用预训练编码器-解码器和大型语言模型,已经设立了新的性能基准。本文展示了多项实验,这些实验复现并扩展了近期基于NMT的SPARQL生成研究,比较了预训练语言模型(PLMs)、非预训练语言模型(NPLMs)和大型语言模型(LLMs),重点考察了问题标注和复制机制的影响,并测试了使用LLMs的各种微调方法。特别是,我们对模型进行了系统性的错误分析,并测试了它们的泛化能力。我们的研究表明,复制机制为大多数PLMs和NPLMs带来了显著的性能提升。数据标注对于生成正确的URI至关重要,其中“tag-within”策略被证明是最有效的方法。此外,我们的发现揭示,错误的主要来源是SPARQL查询中的不正确URI,在使用基础模型时,这些URI有时会被幻觉生成的URI所替代。使用复制机制时不会发生这种情况,但有时会导致在候选项中选择错误的URI。最后,所测试的LLMs的性能未能达到预期结果。

英文摘要:

In recent years, the field of neural machine translation (NMT) for SPARQL query generation has witnessed significant growth. Incorporating the copy mechanism with traditional encoder-decoder architectures and using pre-trained encoder-decoders and large language models have set new performance benchmarks. This paper presents various experiments that replicate and expand upon recent NMT-based SPARQL generation studies, comparing pre-trained language models (PLMs), non-pre-trained language models (NPLMs), and large language models (LLMs), highlighting the impact of question annotation and the copy mechanism and testing various fine-tuning methods using LLMs. In particular, we provide a systematic error analysis of the models and test their generalization ability. Our study demonstrates that the copy mechanism yields significant performance enhancements for most PLMs and NPLMs. Annotating the data is pivotal to generating correct URIs, with the "tag-within" strategy emerging as the most effective approach. Additionally, our findings reveal that the primary source of errors stems from incorrect URIs in SPARQL queries that are sometimes replaced with hallucinated URIs when using base models. This does not happen using the copy mechanism, but it sometimes leads to selecting wrong URIs among candidates. Finally, the performance of the tested LLMs fell short of achieving the desired outcomes.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑