Numerical reasoning is an important task in the analysis of financial documents. It helps in understanding and performing numerical predictions with logical conclusions for the given query seeking answers from financial texts. Recently, Large Language Models (LLMs) have shown promising results in multiple Question-Answering (Q-A) systems with the capability of logical reasoning. As documents related to finance often consist of long and complex financial contexts, LLMs appear well-suited for building high-quality automated financial question-answering systems. However, LLMs often face challenges in accurately processing the various numbers within financial reports. Extracting numerical data from unstructured text and semi-structured tables, and reliably performing accurate calculations, remains a significant bottleneck for numerical reasoning in most state-of-the-art LLMs. Recent studies have shown that structured data augmentations, such as Knowledge Graphs (KGs), have notably improved the predictions of LLMs along with logical explanations. Thus, it is an important requirement to consider inherent structured information in financial reports while using LLMs for various financial analytics. This paper proposes a framework to incorporate structured information using KGs along with LLM predictions for numerical reasoning tasks. The KGs are extracted using a proposed schema inherently from the document under processing. We evaluated our proposed framework over the benchmark data FinQA, using an open-source LLM, namely Llama 3.1 8B Instruct. We observed that the proposed framework improved execution accuracy by approximately 12% relative to the vanilla LLM.
Are Large Language Models Reliable Reviewers? A Benchmark for Error Detection in Financial Documents
College of Computer Science and Artificial Intelligence, Fudan University(复旦大学计算机科学与人工智能学院);School of Data Science, Fudan University(复旦大学数据科学学院);Ant Group(蚂蚁集团);School of Information and School of Smart Governance, Renmin University of China(中国人民大学信息学院与智慧治理学院)
Knowledge-Augmented Large Language Model Agents for Explainable Financial Decision-Making
Boston University(波士顿大学);Hofstra University(霍夫斯特拉大学);University of Southern California(南加州大学);Georgia Institute of Technology(佐治亚理工学院);Carnegie Mellon University(卡内基梅隆大学)
Benchmarking Multi-Agent LLM Architectures for Financial Document Processing: A Comparative Study of Orchestration Patterns, Cost-Accuracy Tradeoffs and Production Scaling Strategies
Department of Computer Science and Engineering(计算机科学与工程系);New York University(纽约大学);New York, NY 10012(纽约市NY 10012)
FinAuditing: A Financial Taxonomy-Structured Multi-Document Benchmark for Evaluating LLMs
Columbia University(哥伦比亚大学);Stevens Institute of Technology(史蒂文斯理工学院);Rensselaer Polytechnic Institute(拉特格斯理工学院);University of Montreal(蒙特利尔大学);McGill University(麦吉尔大学);MBZUAI(麦吉尔大学人工智能研究所);Cardiff University(卡迪夫大学);The University of Manchester(曼彻斯特大学);Harvard University(哈佛大学)
Enabling and Analyzing How to Efficiently Extract Information from Hybrid Long Documents with LLMs
School of Software & Microelectronics, Peking University(北京大学软件与微电子学院);Institute of Software Chinese Academy of Sciences(中国科学院软件研究所);Microsoft(微软公司);University of Technology Sydney(悉尼科技大学)
Rethinking Retrieval: From Traditional Retrieval Augmented Generation to Agentic and Non-Vector Reasoning Systems in the Financial Domain for Large Language Models
FinChain: A Symbolic Benchmark for Verifiable Chain-of-Thought Financial Reasoning
MBZUAI;Syllogia;The University of Tokyo(东京大学);The Fin AI(Fin AI);Cornell University(康奈尔大学);The University of Melbourne(墨尔本大学);IIT Delhi(德里理工学院);The University of Manchester(曼彻斯特大学)
FinBoardBench: Benchmarking Dynamic Wealth Management and Strategic Financial Reasoning of LLMs via Board Game Simulations
School of Computer Science and Engineering, Macau University of Science and Technology, Macau, China(1 计算机科学与工程学院,澳门科技大学,澳门,中国);School of Economics, Anhui University, Anhui, China(2 经济学院,安徽大学,安徽,中国);SKLPlanets, Macau University of Science and Technology, Macau, China(3 SKLPlanets,澳门科技大学,澳门,中国);Department of Computer and Information Science, University of Macau, Macau, China(4 计算机与信息科学系,澳门大学,澳门,中国);School of Energy and Power Engineering, Huazhong University of Science and Technology, Hubei, China(5 能源与动力工程学院,华中科技大学,湖北,中国)
Beyond Classification: Financial Reasoning in State-of-the-Art Language Models
Seoul National University of Science and Technology(首尔国立科技大学);Seoul National University(首尔国立大学);MODULABS;Yonsei University(延世大学);OneLineAI;Chung-Ang University(中央大学)
Document-Level Numerical Reasoning across Single and Multiple Tables in Financial Reports
Department of Computer Science and Information Engineering, National Taiwan University(国立台湾大学资讯工程学系);FinTech Center, National Taiwan University(国立台湾大学金融科技中心)
DSC-IITISM at FinCausal 2021: Combining POS tagging with Attention-based Contextual Representations for Identifying Causal Relationships in Financial Documents
Indian Institute of Technology (ISM), Dhanbad(印度理工学院(ISM)达巴德分校)