arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2404.04068cs.CL
收藏

评估信息提取的质量

Assessing the quality of information extraction

Creative Dock(创意码头公司)

Filip Seitl, Tomáš Kovářík, Soheyla Mirshahi, Jan Kryštůfek, Rastislav Dujava, Matúš Ondreička, Herbert Ullrich, Petr Gronat

AI总结:

本文提出一个自动框架,利用评分评估大型语言模型从非结构化数据中提取实体及属性信息的质量与完整性,并处理模型输入输出限制,以应对标注数据稀缺的挑战。

摘要:

大型语言模型的进步显著提升了从非结构化和半结构化数据源中提取信息的效率。随着这些技术成为各种应用不可或缺的组成部分,建立信息提取质量的客观衡量标准变得至关重要。然而,标注数据的稀缺对这一目标构成了重大挑战。在本文中,我们引入了一个自动框架来评估信息提取/检索的质量及其完整性。该框架聚焦于以实体及其属性形式进行的信息提取。我们讨论了如何处理大型语言模型的输入/输出大小限制,并分析了它们在提取信息时的性能。特别是,我们引入了评分来评估提取质量,并提供了关于如何解读这些评分的广泛讨论。

英文摘要:

Advances in large language models have notably enhanced the efficiency of information extraction from unstructured and semi-structured data sources. As these technologies become integral to various applications, establishing an objective measure for the quality of information extraction becomes imperative. However, the scarcity of labeled data presents significant challenges to this endeavor. In this paper, we introduce an automatic framework to assess the quality of the information extraction/retrieval and its completeness. The framework focuses on information extraction in the form of entity and its properties. We discuss how to handle the input/output size limitations of the large language models and analyze their performance when extracting the information. In particular, we introduce scores to evaluate the quality of the extraction and provide an extensive discussion on how to interpret them.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑