arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2602.17072cs.CL
收藏

BankMathBench: 一个用于银行场景数值推理的基准测试

BankMathBench: A Benchmark for Numerical Reasoning in Banking Scenarios

KakaoBank Corp.(Kakao银行公司);Korea Advanced Institute of Science(韩国先进科学研究院)

Yunseung Lee, Subin Kim, Youngjun Kwak, Jaegul Choo

AI总结:

BankMathBench是一个专门针对银行场景的数值推理基准测试,通过多难度层级的任务设计提升LLMs在金融计算中的准确性和推理能力。

摘要:

大型语言模型(LLMs)基于的聊天机器人正在越来越多地被应用于金融领域,特别是在数字银行中,用于处理关于存款、储蓄和贷款等产品的客户咨询。然而,这些模型在核心银行计算方面仍表现出较低的准确性,包括总支付估计、具有不同利率的产品比较以及提前还款条件下的利息计算。此类任务需要多步骤的数值推理和对银行产品的上下文理解,然而现有LLMs往往会出现系统性错误——错误解释产品类型、错误应用条件或在涉及指数和几何级数的基本计算中失败。然而,此类错误很少被现有基准测试所捕捉。数学数据集专注于基础数学问题,而金融基准主要针对金融文档,使日常银行场景研究不足。为了解决这一限制,我们提出了BankMathBench,一个专门领域的数据集,反映了真实的银行任务。BankMathBench分为三个难度级别——基础、中级和高级,分别对应单一产品推理、多产品比较和多条件场景。当在BankMathBench上训练时,开源LLMs在公式生成和数值推理准确性方面表现出显著的改进,证明了该数据集在增强领域推理方面的有效性。通过工具增强的微调,模型在基础、中级和高级任务上的平均准确率分别提高了57.6%、75.1%和62.9%,显著超过了零样本基线。这些发现突显了BankMathBench作为评估和推动LLMs在真实银行场景中数值推理能力的可靠基准测试的重要性。

英文摘要:

Large language models (LLMs)-based chatbots are increasingly being adopted in the financial domain, particularly in digital banking, to handle customer inquiries about products such as deposits, savings, and loans. However, these models still exhibit low accuracy in core banking computations-including total payout estimation, comparison of products with varying interest rates, and interest calculation under early repayment conditions. Such tasks require multi-step numerical reasoning and contextual understanding of banking products, yet existing LLMs often make systematic errors-misinterpreting product types, applying conditions incorrectly, or failing basic calculations involving exponents and geometric progressions. However, such errors have rarely been captured by existing benchmarks. Mathematical datasets focus on fundamental math problems, whereas financial benchmarks primarily target financial documents, leaving everyday banking scenarios underexplored. To address this limitation, we propose BankMathBench, a domain-specific dataset that reflects realistic banking tasks. BankMathBench is organized in three levels of difficulty-basic, intermediate, and advanced-corresponding to single-product reasoning, multi-product comparison, and multi-condition scenarios, respectively. When trained on BankMathBench, open-source LLMs exhibited notable improvements in both formula generation and numerical reasoning accuracy, demonstrating the dataset's effectiveness in enhancing domain-specific reasoning. With tool-augmented fine-tuning, the models achieved average accuracy increases of 57.6%p (basic), 75.1%p (intermediate), and 62.9%p (advanced), representing significant gains over zero-shot baselines. These findings highlight BankMathBench as a reliable benchmark for evaluating and advancing LLMs' numerical reasoning in real-world banking scenarios.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑