arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2501.04277physics.comp-phcond-mat.mtrl-sci
收藏

探索大型语言模型在材料科学与冶金工程中的专业能力

Exploring the Expertise of Large Language Models in Materials Science and Metallurgical Engineering

Christophe Bajan, Guillaume Lambard

AI总结:

本文用 MaScQA 基准评测15种 LLM 的材料科学与冶金工程问答能力,发现闭源模型准确率约84%、领先开源模型,并为后续专业化训练与微调建立基线。

摘要:

人工智能在各个领域的融合正在迅速增加,大型语言模型(LLM)在众多应用中也变得愈发普遍。本工作隶属于一个总体项目,该项目旨在专门训练面向材料科学领域的 LLM。为评估这种专业化训练的影响,必须先确立现有 LLM 在材料科学中的基线性能。在本研究中,我们使用 MaScQA 问答(Q&A)基准评估了 15 种不同的 LLM。该基准包含来自工程学研究生能力测试(GATE)的问题,专门用于测试模型回答材料科学与冶金工程相关问题的能力。结果表明,Claude-3.5-Sonnet 和 GPT-4 等闭源 LLM 表现最佳,总体准确率约为 84%;而开源模型 Llama3-70b 和 Phi3-14b 最高分别约为 56% 和 43%。这些发现为 LLM 在应用于材料科学的问答任务上的原始能力提供了基线,并强调通过提示工程和微调策略可为开源模型带来的显著提升。我们预计本工作将推动 LLM 作为材料科学中有价值助手的应用,展示其在这一专业领域及相关子领域中的效用。

英文摘要:

The integration of artificial intelligence into various domains is rapidly increasing, with Large Language Models (LLMs) becoming more prevalent in numerous applications. This work is included in an overall project which aims to train an LLM specifically in the field of materials science. To assess the impact of this specialized training, it is essential to establish the baseline performance of existing LLMs in materials science. In this study, we evaluated 15 different LLMs using the MaScQA question answering (Q&A) benchmark. This benchmark comprises questions from the Graduate Aptitude Test in Engineering (GATE), tailored to test models' capabilities in answering questions related to materials science and metallurgical engineering. Our results indicate that closed-source LLMs, such as Claude-3.5-Sonnet and GPT-4, perform the best with an overall accuracy of ~84%, while the open-source models, Llama3-70b and Phi3-14b, top at ~56% and ~43%, respectively. These findings provide a baseline for the raw capabilities of LLMs on Q&A tasks applied to materials science, and emphasize the substantial improvement that could be brought to open-source models via prompt engineering and fine-tuning strategies. We anticipate that this work could push the adoption of LLMs as valuable assistants in materials science, demonstrating their utility in this specialized domain and related sub-domains.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑