arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2407.06423cs.AI
收藏

InsightBench:通过多步洞察生成评估商业分析智能体

InsightBench: Evaluating Business Analytics Agents Through Multi-Step Insight Generation

ServiceNow Research(ServiceNow研究院);Mila - Quebec AI Institute(米拉-魁北克人工智能研究所);École de Technologie Supérieure(高等技术学院);University of British Columbia(不列颠哥伦比亚大学);University of Waterloo(滑铁卢大学);University of Victoria(维多利亚大学)
Gaurav Sahu, Abhay Puri, Juan Rodriguez, Amirhossein Abaskohi, Mohammad Chegini, Alexandre Drouin, Perouz Taslakian, Valentina Zantedeschi, Alexandre Lacoste, D… 展开作者

Gaurav Sahu, Abhay Puri, Juan Rodriguez, Amirhossein Abaskohi, Mohammad Chegini, Alexandre Drouin, Perouz Taslakian, Valentina Zantedeschi, Alexandre Lacoste, David Vazquez, Nicolas Chapados, Christopher Pal, Sai Rajeswar Mudumba, Issam Hadj Laradji

AI总结:

InsightBench是一个包含100个商业数据集的基准,用于评估智能体端到端数据分析能力,其提出的AgentPoirot智能体在洞察生成上优于现有方法,并利用LLaMA-3进行双向评估。

摘要:

数据分析对于从数据中提取有价值的洞察至关重要,这些洞察能够帮助组织做出有效决策。我们引入了InsightBench,一个具有三个关键特征的基准数据集。首先,它包含100个代表多样化商业用例(如金融和事件管理)的数据集,每个数据集都附带一组精心策划并植入数据集中的洞察。其次,与现有专注于回答单一查询的基准不同,InsightBench根据智能体执行端到端数据分析的能力来评估它们,包括提出问题、解释答案以及生成洞察摘要和可操作步骤。第三,我们进行了全面的质量保证,以确保基准中的每个数据集都有明确的目标,并包含相关且有意义的提问和分析。此外,我们实现了一种双向评估机制,使用LLaMA-3作为有效的开源评估器来评估智能体提取洞察的能力。我们还提出了AgentPoirot,这是我们能够执行端到端数据分析的基线数据分析智能体。我们在InsightBench上的评估显示,AgentPoirot优于现有专注于解决单一查询的方法(如Pandas Agent)。我们还比较了开源和闭源大语言模型以及各种评估策略的性能。总体而言,该基准作为一个测试平台,以激励在全面自动化数据分析方面的进一步发展,可在此访问:https://github.com/ServiceNow/insight-bench。

英文摘要:

Data analytics is essential for extracting valuable insights from data that can assist organizations in making effective decisions. We introduce InsightBench, a benchmark dataset with three key features. First, it consists of 100 datasets representing diverse business use cases such as finance and incident management, each accompanied by a carefully curated set of insights planted in the datasets. Second, unlike existing benchmarks focusing on answering single queries, InsightBench evaluates agents based on their ability to perform end-to-end data analytics, including formulating questions, interpreting answers, and generating a summary of insights and actionable steps. Third, we conducted comprehensive quality assurance to ensure that each dataset in the benchmark had clear goals and included relevant and meaningful questions and analysis. Furthermore, we implement a two-way evaluation mechanism using LLaMA-3 as an effective, open-source evaluator to assess agents' ability to extract insights. We also propose AgentPoirot, our baseline data analysis agent capable of performing end-to-end data analytics. Our evaluation on InsightBench shows that AgentPoirot outperforms existing approaches (such as Pandas Agent) that focus on resolving single queries. We also compare the performance of open- and closed-source LLMs and various evaluation strategies. Overall, this benchmark serves as a testbed to motivate further development in comprehensive automated data analytics and can be accessed here: https://github.com/ServiceNow/insight-bench.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑