arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2406.06589cs.CLcs.AI
收藏

PatentEval:理解专利生成中的错误

PatentEval: Understanding Errors in Patent Generation

Inria(法国国家信息与自动化研究所);Qatent;LISN;University Paris-Saclay(巴黎萨克雷大学)

You Zuo, Kim Gerdes, Eric Villemonte de La Clergerie, Benoît Sagot

AI总结:

本文提出针对专利文本生成的综合错误类型体系与基准PatentEval,通过人类标注比较专利专用模型与通用大语言模型,并评估指标与专家判断的一致性,揭示当前模型在该领域的能力与局限。

摘要:

在这项工作中,我们引入了一种专门设计的综合错误类型体系,用于评估机器生成专利文本中的两项不同任务:从权利要求生成摘要,以及基于先前权利要求生成下一个权利要求。我们还开发了一个基准PatentEval,用于在此背景下系统评估语言模型。我们的研究包括由人类标注的各种模型的比较分析,范围从在训练期间专门针对专利领域任务进行适配的模型,到最新的通用大语言模型(LLMs)。此外,我们探索并评估了一些指标,以在专利文本评估中近似人类判断,分析这些指标与专家评估的一致程度。这些方法为当前语言模型在专利文本生成专业领域的能力和局限性提供了有价值的见解。

英文摘要:

In this work, we introduce a comprehensive error typology specifically designed for evaluating two distinct tasks in machine-generated patent texts: claims-to-abstract generation, and the generation of the next claim given previous ones. We have also developed a benchmark, PatentEval, for systematically assessing language models in this context. Our study includes a comparative analysis, annotated by humans, of various models. These range from those specifically adapted during training for tasks within the patent domain to the latest general-purpose large language models (LLMs). Furthermore, we explored and evaluated some metrics to approximate human judgments in patent text evaluation, analyzing the extent to which these metrics align with expert assessments. These approaches provide valuable insights into the capabilities and limitations of current language models in the specialized field of patent text generation.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑