SciText2Eq: 评估大语言模型在科学创造力中的可解释方程生成
SciText2Eq: Assessing LLMs for Explainable Equation Generation for Scientific Creativity
AI总结：
研究大语言模型从科学文本生成数学方程的能力，构建AI论文数据集，提出可解释方程生成流程，并设计结合自动指标、LLM评估和人工判断的评估协议，发现LLM在语义准确性上表现不足。
摘要：
本文研究了大语言模型（LLMs）从科学文本生成数学方程的能力。先前的工作面临非结构化基础、多方程依赖和人类对齐评估的挑战。为此，我们构建了一个AI研究论文数据集，将上下文段落与真实方程和变量描述配对。我们开发了一个可解释的方程生成流程，并在多种开源和闭源LLM骨干上进行了评估。我们引入了一个评估协议，结合自动指标、基于LLM的评分标准和人工判断，以评估准确性、可解释性和人机对齐。结果表明，LLM在基于词汇和句法的相似性上表现中等，但在语义准确性上存在困难。基于LLM的评估与人工判断之间的比较显示对齐有限，凸显了使用LLM评估方程质量的挑战。这些发现为改进方程生成模型和开发更可靠的科学文本评估方法提供了见解。我们提供了代码和数据以供复现。
英文摘要：
This work investigates the ability of large language models (LLMs) to generate mathematical equations from scientific texts. Prior work faces challenges in unstructured grounding, multi-equation dependency, and humanaligned evaluation. To this end, we construct a dataset of AI research papers, pairing contextual passages with ground-truth equations and variable descriptions. We develop an explainable equation generation workflow and evaluate it across diverse open- and closed-source LLM backbones. We introduce an evaluation protocol combining automatic metrics, LLM-based rubrics, and human judgments to assess accuracy, explainability, and human-LLM alignment. Results indicate that LLMs perform moderately on lexical- and syntactic-based similarity, while struggling with semantic accuracy. Comparisons between LLM-based evaluations and human judgments reveal limited alignment, highlighting challenges in using LLMs to assess equation quality. These findings offer insights for improving equation generation models and developing more reliable evaluation methods for scientific text. We provide code and data for reproducibility.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
