In this work, we introduce a comprehensive error typology specifically designed for evaluating two distinct tasks in machine-generated patent texts: claims-to-abstract generation, and the generation of the next claim given previous ones. We have also developed a benchmark, PatentEval, for systematically assessing language models in this context. Our study includes a comparative analysis, annotated by humans, of various models. These range from those specifically adapted during training for tasks within the patent domain to the latest general-purpose large language models (LLMs). Furthermore, we explored and evaluated some metrics to approximate human judgments in patent text evaluation, analyzing the extent to which these metrics align with expert assessments. These approaches provide valuable insights into the capabilities and limitations of current language models in the specialized field of patent text generation.
MoZIP: A Multilingual Benchmark to Evaluate Large Language Models in Intellectual Property
Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences(中国科学院深圳先进技术研究院);Shenzhen Technology University(深圳技术大学);Shenzhen University(深圳大学);Harbin Institute of Technology, Shenzhen(哈尔滨工业大学(深圳))
Self-Knowledge Retrieval Augmented Generation Framework for Patent Matching
School of Computer Science and Technology, Zhejiang University(浙江大学计算机科学与技术学院);ZJU-UIUC Institute, Zhejiang University(浙江大学ZJU-UIUC学院);Shaoxing K3i Technology Co. Ltd(绍兴K3i科技有限公司);State Key Laboratory of CAD&CG, Zhejiang University(浙江大学CAD&CG国家重点实验室)
IPEval: A Bilingual Intellectual Property Agency Consultation Evaluation Benchmark for Large Language Models
School of Future Technology, Dalian University of Technology(大连理工大学未来技术学院);School of Computer Science and Technology, Dalian University of Technology(大连理工大学计算机科学与技术学院);School of Public Administration and Policy, Dalian University of Technology(大连理工大学公共管理与政策学院)
IPBench: Benchmarking the Knowledge of Large Language Models in Intellectual Property
Shenzhen Key Laboratory for High Performance Data Mining, Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences, China(深圳高性能数据挖掘重点实验室,深圳先进技术研究院,中国科学院,中国);University of Chinese Academy of Sciences, China(中国科学院大学,中国);Dalian University of Technology, China(大连理工大学,中国);School of Biomedical Engineering, UNSW Sydney, Australia(生物医学工程学院,新南威尔士大学悉尼分校,澳大利亚);Shenzhen MSU-BIT University, China(深圳MSU-BIT大学,中国);Shenzhen University of Advanced Technology, China(深圳大学先进技术学院,中国)