MaterialBENCH：评估大型语言模型的大学水平材料科学问题解决能力
MaterialBENCH: Evaluating College-Level Materials Science Problem-Solving Abilities of Large Language Models
AI总结：
本文构建了大学水平材料科学基准数据集MaterialBENCH，包含自由响应和多项选择两类问题，通过实验分析不同LLMs的性能差异及系统提示的影响，旨在推动LLMs推理能力发展以助力材料研究。
摘要：
本文构建了一个面向大型语言模型（LLMs）的大学水平材料科学领域基准数据集MaterialBENCH。该数据集基于大学教材，包含问题-答案对，分为两种类型：自由响应型和多项选择型。多项选择型问题通过为正确答案添加三个错误答案作为选项构建，使LLMs可从四个选项中选择一个作答。除答案格式外，自由响应型和多项选择型的大部分问题内容重叠。我们使用MaterialBENCH在LLMs上进行实验，包括ChatGPT-3.5、ChatGPT-4、Bard（实验时版本）以及通过OpenAI API调用的GPT-3.5和GPT-4。分析并讨论了LLMs在MaterialBENCH上表现的异同，还研究了同一模型在自由响应型与多项选择型问题上的性能差异，以及系统提示对多项选择型问题的影响。我们期望MaterialBENCH能推动LLMs推理能力的进一步发展，以解决更复杂的问题，并最终为材料研究与发现做出贡献。
英文摘要：
A college-level benchmark dataset for large language models (LLMs) in the materials science field, MaterialBENCH, is constructed. This dataset consists of problem-answer pairs, based on university textbooks. There are two types of problems: one is the free-response answer type, and the other is the multiple-choice type. Multiple-choice problems are constructed by adding three incorrect answers as choices to a correct answer, so that LLMs can choose one of the four as a response. Most of the problems for free-response answer and multiple-choice types overlap except for the format of the answers. We also conduct experiments using the MaterialBENCH on LLMs, including ChatGPT-3.5, ChatGPT-4, Bard (at the time of the experiments), and GPT-3.5 and GPT-4 with the OpenAI API. The differences and similarities in the performance of LLMs measured by the MaterialBENCH are analyzed and discussed. Performance differences between the free-response type and multiple-choice type in the same models and the influence of using system massages on multiple-choice problems are also studied. We anticipate that MaterialBENCH will encourage further developments of LLMs in reasoning abilities to solve more complicated problems and eventually contribute to materials research and discovery.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
