arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2408.12188cs.CLcs.AI
收藏

利用大语言模型对结构化数据中的事实知识进行推理

Reasoning Factual Knowledge in Structured Data with Large Language Models

Hong Kong Polytechnic University(香港理工大学);University Technology Sydney(悉尼科技大学);The Hong Kong University of Science and Technology (Guangzhou)(香港科技大学(广州));Soochow University(苏州大学);The Education University of Hong Kong(香港教育大学)

Sirui Huang, Yanggan Gu, Xuming Hu, Zhonghao Li, Qing Li, Guandong Xu

AI总结:

本文提出基准StructFact,包含8,340个事实问题,用于评估大语言模型在结构化数据中进行事实知识推理的能力,实验揭示了当前模型在此方面的局限性。

摘要:

大语言模型(LLMs)在各种自然语言处理任务中取得了显著进展,这得益于其理解和推理事实知识的能力。然而,大量事实知识存储在结构化数据中,其具有与预训练所用的非结构化文本不同的独特特征。这种差异会引入难以察觉的推理参数偏差,给LLMs有效利用和推理结构化数据以准确推断事实知识带来挑战。为此,我们提出了一个名为StructFact的基准,以评估LLMs在推断事实知识时的结构推理能力。StructFact包含8,340个事实性问题,涵盖各种任务、领域、时间线和地区。该基准使我们能够研究LLMs在源自结构事实独特特征的五项事实任务上的能力。对采用不同训练策略的一组LLMs进行的广泛实验揭示了当前LLMs在从结构化数据中推断事实知识方面的局限性。我们将此基准作为指南针,以导航LLMs在知识敏感任务中推理结构化数据的优缺点,并鼓励相关实际应用的进步。请在https://github.com/EganGu/StructFact获取我们的代码。

英文摘要:

Large language models (LLMs) have made remarkable progress in various natural language processing tasks as a benefit of their capability to comprehend and reason with factual knowledge. However, a significant amount of factual knowledge is stored in structured data, which possesses unique characteristics that differ from the unstructured texts used for pretraining. This difference can introduce imperceptible inference parameter deviations, posing challenges for LLMs in effectively utilizing and reasoning with structured data to accurately infer factual knowledge. To this end, we propose a benchmark named StructFact, to evaluate the structural reasoning capabilities of LLMs in inferring factual knowledge. StructFact comprises 8,340 factual questions encompassing various tasks, domains, timelines, and regions. This benchmark allows us to investigate the capability of LLMs across five factual tasks derived from the unique characteristics of structural facts. Extensive experiments on a set of LLMs with different training strategies reveal the limitations of current LLMs in inferring factual knowledge from structured data. We present this benchmark as a compass to navigate the strengths and weaknesses of LLMs in reasoning with structured data for knowledge-sensitive tasks, and to encourage advancements in related real-world applications. Please find our code at https://github.com/EganGu/StructFact.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑