arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2601.07754cs.CL
收藏

先结构,再推理:利用知识图谱增强大型语言模型进行财务文档中的数值推理

Structure First, Reason Next: Enhancing a Large Language Model using Knowledge Graph for Numerical Reasoning in Financial Documents

Aryan Mishra, Akash Anil

AI总结:

本文提出利用知识图谱增强大型语言模型,以提高财务文档中的数值推理能力,实验表明该方法在准确性上提升了约12%。

摘要:

数值推理是财务文档分析中的重要任务。它有助于理解并基于给定查询从财务文本中进行数值预测,得出逻辑结论。最近,大型语言模型(LLMs)在多个问答(Q-A)系统中展示了在逻辑推理方面有前景的结果。由于与金融相关的文档通常包含长而复杂的财务背景,LLMs似乎非常适合构建高质量的自动化金融问答系统。然而,LLMs在准确处理财务报告中的各种数字时常常面临挑战。从无结构文本和半结构化表格中提取数值数据,并可靠地进行准确计算,仍然是大多数最新LLMs在数值推理中的一大瓶颈。最近的研究表明,结构化数据增强,如知识图谱(KGs),已显著提高了LLMs的预测能力以及逻辑解释。因此,在使用LLMs进行各种金融分析时,考虑财务报告中固有的结构化信息是一个重要的要求。本文提出了一种框架,通过KGs结合LLM预测来纳入结构化信息,用于数值推理任务。KGs是通过从正在处理的文档中提出的模式进行提取。我们通过基准数据FinQA评估了所提出的框架,使用开源LLM,即Llama 3.1 8B Instruct。我们观察到,所提出的框架相对于普通的LLM在执行准确性上提高了大约12%。

英文摘要:

Numerical reasoning is an important task in the analysis of financial documents. It helps in understanding and performing numerical predictions with logical conclusions for the given query seeking answers from financial texts. Recently, Large Language Models (LLMs) have shown promising results in multiple Question-Answering (Q-A) systems with the capability of logical reasoning. As documents related to finance often consist of long and complex financial contexts, LLMs appear well-suited for building high-quality automated financial question-answering systems. However, LLMs often face challenges in accurately processing the various numbers within financial reports. Extracting numerical data from unstructured text and semi-structured tables, and reliably performing accurate calculations, remains a significant bottleneck for numerical reasoning in most state-of-the-art LLMs. Recent studies have shown that structured data augmentations, such as Knowledge Graphs (KGs), have notably improved the predictions of LLMs along with logical explanations. Thus, it is an important requirement to consider inherent structured information in financial reports while using LLMs for various financial analytics. This paper proposes a framework to incorporate structured information using KGs along with LLM predictions for numerical reasoning tasks. The KGs are extracted using a proposed schema inherently from the document under processing. We evaluated our proposed framework over the benchmark data FinQA, using an open-source LLM, namely Llama 3.1 8B Instruct. We observed that the proposed framework improved execution accuracy by approximately 12% relative to the vanilla LLM.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑