ChemPro:一种渐进式的化学基准测试用于大语言模型
ChemPro: A Progressive Chemistry Benchmark for Large Language Models
AI总结:
ChemPro通过4100个化学问题-答案对评估大语言模型在化学领域的表现,涵盖多个难度层级,揭示了LLM在科学推理中的局限性。
摘要:
我们介绍了ChemPro,一个包含4100个自然语言问题-答案对的渐进式基准测试,涵盖四个难度递增的章节,旨在评估大语言模型(LLMs)在广泛化学主题中的能力。问题类型包括选择题和数值题,分布在细粒度信息回忆、长周期推理、多概念问题、需要细致表达的问题解决以及基础问题中,以平衡比例覆盖生物化学、无机化学、有机化学和物理化学。ChemPro的设计类似于学生在基础到高中化学课程中的学术评估。问题难度逐步增加,严格测试LLM从解决基础问题到解决更复杂挑战的能力。我们评估了45种以上的最新LLM,涵盖开源和专有变体,分析表明虽然LLM在基础化学问题上表现良好,但其准确性在不同问题类型和复杂性水平上下降。这些发现突显了LLM在一般科学推理和理解中的关键限制,并指出了未被研究的难度维度,强调了需要更稳健的方法来改进LLM。
英文摘要:
We introduce ChemPro, a progressive benchmark with 4100 natural language question-answer pairs in Chemistry, across 4 coherent sections of difficulty designed to assess the proficiency of Large Language Models (LLMs) in a broad spectrum of general chemistry topics. We include Multiple Choice Questions and Numerical Questions spread across fine-grained information recall, long-horizon reasoning, multi-concept questions, problem-solving with nuanced articulation, and straightforward questions in a balanced ratio, effectively covering Bio-Chemistry, Inorganic-Chemistry, Organic-Chemistry and Physical-Chemistry. ChemPro is carefully designed analogous to a student's academic evaluation for basic to high-school chemistry. A gradual increase in the question difficulty rigorously tests the ability of LLMs to progress from solving basic problems to solving more sophisticated challenges. We evaluate 45+7 state-of-the-art LLMs, spanning both open-source and proprietary variants, and our analysis reveals that while LLMs perform well on basic chemistry questions, their accuracy declines with different types and levels of complexity. These findings highlight the critical limitations of LLMs in general scientific reasoning and understanding and point towards understudied dimensions of difficulty, emphasizing the need for more robust methodologies to improve LLMs.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。