arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2401.11052cs.CL
收藏

使用大型语言模型从材料科学文献中挖掘实验数据:一项评估研究

Mining experimental data from Materials Science literature with Large Language Models: an evaluation study

National Institute for Materials Science(物质材料研究机构);University of Tsukuba(筑波大学)

Luca Foppiano, Guillaume Lambard, Toshiyuki Amagasa, Masashi Ishii

AI总结:

本研究评估了GPT-3.5-Turbo、GPT-4等LLMs在材料科学文献中进行命名实体识别和关系抽取的能力,发现微调后的GPT-3.5-Turbo在关系抽取上超越基线,但专用模型在提取复杂实体上仍优于LLMs。

摘要:

本研究旨在评估诸如GPT-3.5-Turbo、GPT-4和GPT-4-Turbo等大型语言模型(LLMs)从材料科学科学文档中提取结构化信息的能力。为此,我们主要关注信息提取的两个关键任务:(i)所研究材料和物理性质的命名实体识别(NER)以及(ii)这些实体之间的关系抽取(RE)。由于材料信息学(MI)领域明显缺乏数据集,我们使用了基于超导体研究的SuperMat和通用测量评估语料库MeasEval进行评估。LLMs执行这些任务的性能与基于BERT架构的传统模型和基于规则的方法(基线)进行了基准比较。我们引入了一种用于复杂材料表达式比较分析的新颖方法,强调化学式的标准化,以应对材料科学信息评估中固有的复杂性。对于NER,LLMs在零样本提示下未能超越基线,并且在少样本提示下仅表现出有限的改进。然而,采用适当策略进行微调的GPT-3.5-Turbo在RE任务上超越了所有模型,包括基线。在没有任何微调的情况下,GPT-4和GPT-4-Turbo在仅提供几个示例后就展现出卓越的推理和关系抽取能力,超越了基线。总体而言,结果表明,尽管LLMs在连接概念方面展示了相关的推理技能,但对于需要提取像材料这样复杂领域特定实体的任务,专用模型目前是更好的选择。这些见解为未来工作中适用于其他材料科学子领域提供了初步指导。

英文摘要:

This study is dedicated to assessing the capabilities of large language models (LLMs) such as GPT-3.5-Turbo, GPT-4, and GPT-4-Turbo in extracting structured information from scientific documents in materials science. To this end, we primarily focus on two critical tasks of information extraction: (i) a named entity recognition (NER) of studied materials and physical properties and (ii) a relation extraction (RE) between these entities. Due to the evident lack of datasets within Materials Informatics (MI), we evaluated using SuperMat, based on superconductor research, and MeasEval, a generic measurement evaluation corpus. The performance of LLMs in executing these tasks is benchmarked against traditional models based on the BERT architecture and rule-based approaches (baseline). We introduce a novel methodology for the comparative analysis of intricate material expressions, emphasising the standardisation of chemical formulas to tackle the complexities inherent in materials science information assessment. For NER, LLMs fail to outperform the baseline with zero-shot prompting and exhibit only limited improvement with few-shot prompting. However, a GPT-3.5-Turbo fine-tuned with the appropriate strategy for RE outperforms all models, including the baseline. Without any fine-tuning, GPT-4 and GPT-4-Turbo display remarkable reasoning and relationship extraction capabilities after being provided with merely a couple of examples, surpassing the baseline. Overall, the results suggest that although LLMs demonstrate relevant reasoning skills in connecting concepts, specialised models are currently a better choice for tasks requiring extracting complex domain-specific entities like materials. These insights provide initial guidance applicable to other materials science sub-domains in future work.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑