利用对话式语言模型与提示工程从研究论文中提取准确材料数据
Extracting Accurate Materials Data from Research Papers with Conversational Language Models and Prompt Engineering
AI总结：
本文提出 ChatExtract 方法，通过工程化提示和后续问答机制，利用对话式大语言模型以最少人工投入实现材料数据的高精度自动提取，精确率与召回率接近 90%，并成功构建了金属玻璃和高熵合金数据库。
摘要：
目前，人们越来越多地尝试用基于自然语言处理、语言模型以及近期出现的大语言模型（LLMs）的自动化数据提取来替代从研究论文中人工提取数据。尽管这些方法能够高效地从大量研究论文中提取数据，但它们需要大量的前期投入、专业知识和编程工作。在这项工作中，我们提出了 ChatExtract 方法，该方法利用先进的对话式 LLM，能够以最少的前期工作和背景知识完全自动化地实现高精度数据提取。ChatExtract 由一组应用于对话式 LLM 的工程化提示组成，这些提示既能识别包含数据的句子、提取数据，又能通过一系列后续问题确保数据的正确性。这些后续问题在很大程度上克服了 LLM 提供事实不准确回答的已知问题。ChatExtract 可应用于任何对话式 LLM，并能产生高质量的数据提取结果。在材料数据的测试中，我们发现最佳的对话式 LLM（如 ChatGPT-4）的精确率和召回率均接近 90%。我们证明，这种卓越性能得益于对话模型中的信息保留能力，结合有目的的冗余设计以及通过后续提示引入不确定性。这些结果表明，类似 ChatExtract 的方法因其简单性、可迁移性和准确性，很可能在不久的将来成为强大的数据提取工具。最后，我们利用 ChatExtract 开发了金属玻璃临界冷却速率和高熵合金屈服强度的数据库。
英文摘要：
There has been a growing effort to replace manual extraction of data from research papers with automated data extraction based on natural language processing, language models, and recently, large language models (LLMs). Although these methods enable efficient extraction of data from large sets of research papers, they require a significant amount of up-front effort, expertise, and coding. In this work we propose the ChatExtract method that can fully automate very accurate data extraction with minimal initial effort and background, using an advanced conversational LLM. ChatExtract consists of a set of engineered prompts applied to a conversational LLM that both identify sentences with data, extract that data, and assure the data's correctness through a series of follow-up questions. These follow-up questions largely overcome known issues with LLMs providing factually inaccurate responses. ChatExtract can be applied with any conversational LLMs and yields very high quality data extraction. In tests on materials data we find precision and recall both close to 90% from the best conversational LLMs, like ChatGPT-4. We demonstrate that the exceptional performance is enabled by the information retention in a conversational model combined with purposeful redundancy and introducing uncertainty through follow-up prompts. These results suggest that approaches similar to ChatExtract, due to their simplicity, transferability, and accuracy are likely to become powerful tools for data extraction in the near future. Finally, databases for critical cooling rates of metallic glasses and yield strengths of high entropy alloys are developed using ChatExtract.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
