面向材料合成的大语言模型基准测试：原子层沉积案例
Benchmarking large language models for materials synthesis: the case of atomic layer deposition
AI总结：
本文提出面向原子层沉积材料合成的开放式问答基准 ALDbench，并以 GPT-4o 评估其回答质量、相关性与准确性，揭示幻觉及多维度评测的必要性。
摘要：
在这项工作中，我们引入了一个开放式问答基准 ALDbench，用于评估大语言模型（LLM）在材料合成中的表现，尤其是在原子层沉积领域；原子层沉积是一种用于能源应用和微电子领域的薄膜生长技术。我们的基准包含难度从研究生水平到熟悉该领域最新进展的领域专家水平的问题。人类专家依据难度和具体性标准审阅问题，并依据四个不同标准评审模型回答：总体质量、具体性、相关性和准确性。我们在 OpenAI 的 GPT-4o 的一个实例上运行了该基准。模型回答在 1 到 5 的量表上获得了 3.7 的综合质量分，与及格成绩相符。然而，36% 的问题至少获得了一个低于平均水平的分数。对回答的深入分析识别出至少五个疑似幻觉实例。最后，我们观察到问题难度与回答质量、问题难度与回答相关性，以及问题具体性与人类专家评定的回答准确性之间存在具有统计显著性的相关性。这强调了需要在难度或准确性之外，依据多个标准评估 LLM。
英文摘要：
In this work we introduce an open-ended question benchmark, ALDbench, to evaluate the performance of large language models (LLMs) in materials synthesis, and in particular in the field of atomic layer deposition, a thin film growth technique used in energy applications and microelectronics. Our benchmark comprises questions with a level of difficulty ranging from graduate level to domain expert current with the state of the art in the field. Human experts reviewed the questions along the criteria of difficulty and specificity, and the model responses along four different criteria: overall quality, specificity, relevance, and accuracy. We ran this benchmark on an instance of OpenAI's GPT-4o. The responses from the model received a composite quality score of 3.7 on a 1 to 5 scale, consistent with a passing grade. However, 36% of the questions received at least one below average score. An in-depth analysis of the responses identified at least five instances of suspected hallucination. Finally, we observed statistically significant correlations between the difficulty of the question and the quality of the response, the difficulty of the question and the relevance of the response, and the specificity of the question and the accuracy of the response as graded by the human experts. This emphasizes the need to evaluate LLMs across multiple criteria beyond difficulty or accuracy.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
