arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2412.03589cs.AIcs.CLcs.HC
收藏

利用大语言模型进行文本中过程知识图谱提取的人类评估

Human Evaluation of Procedural Knowledge Graph Extraction from Text with Large Language Models

Princeton University(普林斯顿大学);Springer Heidelberg(斯普林格海德堡);ABC Institute(ABC研究所)

Valentina Anita Carriero, Antonia Azzini, Ilaria Baroni, Mario Scrocca, Irene Celino

AI总结:

本文提出利用大语言模型和提示工程方法,从文本中提取过程知识并构建知识图谱,通过用户研究评估其质量和实用性。

摘要:

过程知识是以执行某些任务所需的步骤序列形式表达的技能。通常通过自然语言文本描述,如食谱或维护手册,可能分布在不同文档和系统中,其解释和后续执行通常由读者自行完成。将此类过程表示为知识图谱(KG)可以成为构建数字工具的基础,以支持需要应用或执行这些过程的用户。在本文中,我们利用大语言模型(LLM)的能力,提出了一种提示工程方法,从文本过程提取步骤、动作、对象、设备和时间信息,以根据预定义的本体填充过程知识图谱。我们通过用户研究评估图谱提取结果,以定性和定量评估LLM提取过程知识的感知质量和实用性。我们展示了LLM可以产生可接受质量的输出,并评估了人类评估者对AI的主观感知。

英文摘要:

Procedural Knowledge is the know-how expressed in the form of sequences of steps needed to perform some tasks. Procedures are usually described by means of natural language texts, such as recipes or maintenance manuals, possibly spread across different documents and systems, and their interpretation and subsequent execution is often left to the reader. Representing such procedures in a Knowledge Graph (KG) can be the basis to build digital tools to support those users who need to apply or execute them. In this paper, we leverage Large Language Model (LLM) capabilities and propose a prompt engineering approach to extract steps, actions, objects, equipment and temporal information from a textual procedure, in order to populate a Procedural KG according to a pre-defined ontology. We evaluate the KG extraction results by means of a user study, in order to qualitatively and quantitatively assess the perceived quality and usefulness of the LLM-extracted procedural knowledge. We show that LLMs can produce outputs of acceptable quality and we assess the subjective perception of AI by human evaluators.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑