arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2402.03848cs.CLcs.AI
收藏

ANLS* -- 一种面向生成式大语言模型的通用文档处理度量标准

ANLS* -- A Universal Document Processing Metric for Generative Large Language Models

DeepOpinion

David Peer, Philemon Schöpf, Volckmar Nebendahl, Alexander Rietzler, Sebastian Stabinger

AI总结:

本文提出ANLS*度量标准,用于评估生成式大语言模型在文档处理任务中的性能,并验证了SFT提示方法在多数情况下优于现有技术,提升最高达10个百分点。

摘要:

传统上,判别式模型是文档分类和信息提取等任务的首选。这些模型做出的预测属于有限数量的预定义类别,便于进行二元真/假评估,并能够直接计算F1分数等指标。然而,生成式大语言模型(GLLMs)的最新进展促使该领域发生转变,因为其增强的零样本能力消除了对下游数据集和计算成本高昂的微调的需求。然而,评估GLLMs面临挑战,因为用于判别式模型的二元真/假评估不适用于GLLMs做出的预测。\n本文提出了一种新的生成式模型度量标准,称为ANLS*,用于评估各种任务,包括信息提取和分类任务。ANLS*度量扩展了现有的ANLS度量,作为直接替代品,并且仍然与先前报告的ANLS分数兼容。本文还提供了使用ANLS*度量对7个不同数据集、20多个不同GLLMs以及3种不同提示方法的评估,证明了所提出度量标准的重要性。\n我们还基准测试了一种生成文档提示的新方法,称为SFT,与其他提示技术(如LATIN)进行比较。在几乎所有情况下,SFT都优于其他技术,并提升了最先进水平,有时提高多达10个百分点。\n源代码可在https://github.com/deepopinion/anls_star_metric获取。

英文摘要:

Traditionally, discriminative models have been the predominant choice for tasks like document classification and information extraction. These models make predictions that fall into a limited number of predefined classes, facilitating a binary true or false evaluation and enabling the direct calculation of metrics such as the F1 score. However, recent advancements in generative large language models (GLLMs) have prompted a shift in the field due to their enhanced zero-shot capabilities, which eliminate the need for a downstream dataset and computationally expensive fine-tuning. However, evaluating GLLMs presents a challenge as the binary true or false evaluation used for discriminative models is not applicable to the predictions made by GLLMs. This paper introduces a new metric for generative models called ANLS* for evaluating a wide variety of tasks, including information extraction and classification tasks. The ANLS* metric extends existing ANLS metrics as a drop-in-replacement and is still compatible with previously reported ANLS scores. An evaluation of 7 different datasets, and more than 20 different GLLMs together with 3 different prompting methods using the ANLS* metric is also provided, demonstrating the importance of the proposed metric. We also benchmark a novel approach to generate prompts for documents, called SFT, against other prompting techniques such as LATIN. In almost all cases, SFT outperforms other techniques and improves the state-of-the-art, sometimes by as much as $10$ percentage points. Sources are available at https://github.com/deepopinion/anls_star_metric

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑