arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2410.21741cs.CLcs.AI
收藏

基于多智能体反思框架的金融问答增强研究

Enhancing Financial Question Answering with a Multi-Agent Reflection Framework

University of Illinois at Chicago(伊利诺伊大学芝加哥分校)

Sorouralsadat Fatemi, Yuheng Hu

AI总结:

本研究提出融合多维度评判智能体的多智能体反思框架,通过对推理步骤和答案的多方面反思增强金融问答能力,显著提升开源LLM性能,可作为大模型的高性价比替代方案。

摘要:

尽管大语言模型(LLM)已在众多自然语言处理(NLP)任务中展现出出色能力,但在金融问答(QA)任务中仍存在不足,尤其是涉及数值推理的场景。近期,基于LLM的多智能体框架在多步推理任务中展现出显著效果,而金融问答任务需要从表格和文本中提取相关信息,再对提取的数据进行数值推理以推导答案,因此多步推理对这类任务至关重要。本研究提出一种融合评判智能体的多智能体框架,该智能体可对每个问题的推理步骤和最终答案进行反思。此外,我们通过增加多个评判智能体来优化系统,每个智能体专注于答案的一个特定方面。实验结果表明,与单智能体推理相比,该框架显著提升了性能:LLaMA3-8B模型的性能平均提升15%,LLaMA3-70B模型平均提升5%。此外,该框架的性能与LLaMA3.1-405B、GPT-4o-mini等更大规模的单智能体LLM相当,部分情况下甚至更优,但略逊于Claude-3.5 Sonnet。总体而言,该框架为增强开源LLM的金融问答能力提供了有效解决方案,是Claude-3.5 Sonnet等更大模型的高性价比替代方案。

英文摘要:

While Large Language Models (LLMs) have shown impressive capabilities in numerous Natural Language Processing (NLP) tasks, they still struggle with financial question answering (QA), particularly when numerical reasoning is required. Recently, LLM-based multi-agent frameworks have demonstrated remarkable effectiveness in multi-step reasoning, which is crucial for financial QA tasks as it involves extracting relevant information from tables and text and then performing numerical reasoning on the extracted data to infer answers. In this study, we propose a multi-agent framework incorporating a critic agent that reflects on the reasoning steps and final answers for each question. Additionally, we enhance our system by adding multiple critic agents, each focusing on a specific aspect of the answer. Our results indicate that this framework significantly improves performance compared to single-agent reasoning, with an average performance increase of 15% for the LLaMA3-8B model and 5% for the LLaMA3-70B model. Furthermore, our framework performs on par with, and in some cases surpasses, larger single-agent LLMs such as LLaMA3.1-405B and GPT-4o-mini, though it falls slightly short compared to Claude-3.5 Sonnet. Overall, our framework presents an effective solution to enhance open-source LLMs for financial QA tasks, offering a cost-effective alternative to larger models like Claude-3.5 Sonnet.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑