arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2402.12692cs.CL
收藏

FormulaReasoning:一个用于基于公式的数值推理的数据集

FormulaReasoning: A Dataset for Formula-Based Numerical Reasoning

State Key Laboratory for Novel Software Technology, Nanjing University, Nanjing, China(南京大学新型软件技术国家重点实验室)

Xiao Li, Bolin Zhu, Kaiwen Shi, Sichen Liu, Yin Zhu, Yiwei Liu, Gong Cheng

AI总结:

FormulaReasoning数据集通过引入基于物理原理的数值推理任务,提供细粒度注释和扩展版本,评估多种推理框架,揭示公式推理中的关键挑战与未来研究方向。

摘要:

物理公式在数值推理中的应用是人类的基本能力。尽管现有数据集通常依赖于隐含的数学知识,但很少明确列出底层的公式。为此,我们引入了FormulaReasoning,一个用于基于公式的数值推理的新基准,包含5,324个需要基于外部物理原理进行计算的问题。我们提供了高质量、细粒度的英文和中文注释,包括公式结构、参数名称、符号、值和单位,这些注释通过人工努力和LLM辅助验证进行编纂。此外,我们还提供了一个综合的公式数据库作为外部知识源。为了进一步挑战模型性能,我们通过将多个问题结合开发了数据集的扩展版本。我们评估了各种架构和方法框架,包括检索增强方法、模块化推理(公式生成、参数提取和计算)以及基于偏好的优化。我们的分析识别了基于公式的推理中的关键挑战,突显了未来方法学进步的重要机会。

英文摘要:

The application of physics formulas is a fundamental human capability in numerical reasoning. While existing datasets often rely on implicit mathematical knowledge, they rarely explicitate the underlying formulas. To address this, we introduce FormulaReasoning, a new benchmark for formula-based numerical reasoning comprising 5,324 questions requiring calculations grounded in external physics principles. We provide high-quality, fine-grained annotations in English and Chinese--including formula structures, parameter names, symbols, values, and units--curated through manual effort and LLM-assisted validation. Additionally, we provide a consolidated formula database as an external knowledge source. To further challenge model performance, we develop an extended version of the dataset by coupling multiple questions. We evaluate various architectural and methodological frameworks, including retrieval-augmented methods, modular reasoning (formula generation, parameter extraction, and calculation), and preference-based optimization. Our analysis identifies critical challenges in formula-based reasoning, highlighting significant opportunities for future methodological advancement.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑