BizBench:面向商业与金融的定量推理基准
BizBench: A Quantitative Reasoning Benchmark for Business and Finance
AI总结:
该研究推出面向商业金融领域的定量推理基准BizBench,含八项任务及三类金融代码生成任务,评估发现大语言模型性能瓶颈在于金融理解不足,凸显该领域推理基准的价值。
摘要:
解答商业与金融领域的问题需要推理能力、精准度以及广泛的技术知识。这些要求叠加在一起,使得大语言模型(LLM)难以应对该领域。我们推出BizBench,这是一个用于评估模型对现实金融问题推理能力的基准。BizBench包含八项定量推理任务,聚焦于通过程序合成(program synthesis)完成金融数据问答(QA)。我们纳入了三项基于新收集并增强的问答数据的金融主题代码生成任务。此外,我们分离出了金融问答所需的推理能力:从金融文本和表格中提取中间值的阅读理解能力,以及理解计算复杂解决方案所需的金融概念与公式的能力。这些任务共同评估模型的金融背景知识、解析金融文档的能力以及用代码解决问题的能力。我们对开源和商用大语言模型开展了深入评估,对比分析了聚焦代码的模型与聚焦语言的模型的表现。我们证明,当前性能瓶颈源于大语言模型有限的商业与金融理解能力,凸显了该领域具有挑战性的定量推理基准的价值。
英文摘要:
Answering questions within business and finance requires reasoning, precision, and a wide-breadth of technical knowledge. Together, these requirements make this domain difficult for large language models (LLMs). We introduce BizBench, a benchmark for evaluating models' ability to reason about realistic financial problems. BizBench comprises eight quantitative reasoning tasks, focusing on question-answering (QA) over financial data via program synthesis. We include three financially-themed code-generation tasks from newly collected and augmented QA data. Additionally, we isolate the reasoning capabilities required for financial QA: reading comprehension of financial text and tables for extracting intermediate values, and understanding financial concepts and formulas needed to calculate complex solutions. Collectively, these tasks evaluate a model's financial background knowledge, ability to parse financial documents, and capacity to solve problems with code. We conduct an in-depth evaluation of open-source and commercial LLMs, comparing and contrasting the behavior of code-focused and language-focused models. We demonstrate that the current bottleneck in performance is due to LLMs' limited business and financial understanding, highlighting the value of a challenging benchmark for quantitative reasoning within this domain.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。