InsightBench: Evaluating Business Analytics Agents Through Multi-Step Insight Generation
ServiceNow Research(ServiceNow研究院);Mila - Quebec AI Institute(米拉-魁北克人工智能研究所);École de Technologie Supérieure(高等技术学院);University of British Columbia(不列颠哥伦比亚大学);University of Waterloo(滑铁卢大学);University of Victoria(维多利亚大学)
Gaurav Sahu, Abhay Puri, Juan Rodriguez, Amirhossein Abaskohi, Mohammad Chegini, Alexandre Drouin, Perouz Taslakian, Valentina Zantedeschi, Alexandre Lacoste, D… 展开作者
Gaurav Sahu, Abhay Puri, Juan Rodriguez, Amirhossein Abaskohi, Mohammad Chegini, Alexandre Drouin, Perouz Taslakian, Valentina Zantedeschi, Alexandre Lacoste, David Vazquez, Nicolas Chapados, Christopher Pal, Sai Rajeswar Mudumba, Issam Hadj Laradji
Data analytics is essential for extracting valuable insights from data that can assist organizations in making effective decisions. We introduce InsightBench, a benchmark dataset with three key features. First, it consists of 100 datasets representing diverse business use cases such as finance and incident management, each accompanied by a carefully curated set of insights planted in the datasets. Second, unlike existing benchmarks focusing on answering single queries, InsightBench evaluates agents based on their ability to perform end-to-end data analytics, including formulating questions, interpreting answers, and generating a summary of insights and actionable steps. Third, we conducted comprehensive quality assurance to ensure that each dataset in the benchmark had clear goals and included relevant and meaningful questions and analysis. Furthermore, we implement a two-way evaluation mechanism using LLaMA-3 as an effective, open-source evaluator to assess agents' ability to extract insights. We also propose AgentPoirot, our baseline data analysis agent capable of performing end-to-end data analytics. Our evaluation on InsightBench shows that AgentPoirot outperforms existing approaches (such as Pandas Agent) that focus on resolving single queries. We also compare the performance of open- and closed-source LLMs and various evaluation strategies. Overall, this benchmark serves as a testbed to motivate further development in comprehensive automated data analytics and can be accessed here: https://github.com/ServiceNow/insight-bench.
A Detailed Study on LLM Biases Concerning Corporate Social Responsibility and Green Supply Chains
Research Center Trustworthy Data Science and Security, University Alliance Ruhr(可信数据科学与安全研究所以及鲁尔大学联盟);Department of Computer Science, University of Duisburg-Essen(杜伊斯堡- Essen大学计算机科学系);Department of Computer Science, Technical University Dortmund(图恩大学计算机科学系);Chair of Mathematical Statistics and Applications in Industry, Technical University Dortmund(工业数学统计与应用教授职位,图恩大学)
FDE-Bench: Evaluating End-to-End Delivery from Underspecified Real-World Business Requests
FM Agent Team, Baidu AI Cloud(百度智能云FM智能体团队);Beijing University of Posts and Telecommunications(北京邮电大学);The Chinese University of Hong Kong(香港中文大学);Tsinghua University(清华大学);Peking University(北京大学);University of the Chinese Academy of Sciences(中国科学院大学);Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所)
Evaluating Conformance Measures in Process Mining using Conformance Propositions (Extended version)
RWTH Aachen University(亚琛工业大学);Eindhoven University of Technology(埃因霍温理工大学);Fraunhofer Institute for Applied Information Technology FIT(弗劳恩霍夫应用信息技术研究所FIT)
PartnerMAS: An LLM Hierarchical Multi-Agent Framework for Business Partner Selection on High-Dimensional Features
University of South Florida(佛罗里达州立大学);McGill University(麦吉尔大学);Purdue University(普渡大学);Lingnan University(岭大大学);University of California, Santa Barbara(加州大学圣巴巴拉分校);Hong Kong Baptist University(香港 Baptist 大学)
AIM-Bench: Evaluating Decision-making Biases of Agentic LLM as Inventory Manager
School of Management and Engineering, Nanjing University(南京大学管理工程学院);School of Robotics and Automation, Nanjing University (Suzhou Campus)(南京大学机器人与自动化学院)