arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2408.09667cs.CL
收藏

BLADE:面向数据驱动科学的语言模型智能体基准测试

BLADE: Benchmarking Language Model Agents for Data-Driven Science

University of Washington(华盛顿大学);UC Berkeley(伯克利大学);New York University(纽约大学);Stanford University(斯坦福大学);University of British Columbia(不列颠哥伦比亚大学);Microsoft(微软);George Washington University(乔治·华盛顿大学)
Ken Gu, Ruoxi Shang, Ruien Jiang, Keying Kuang, Richard-John Lin, Donghe Lyu, Yue Mao, Youran Pan, Teng Wu, Jiaqian Yu, Yikun Zhang, Tianmai M. Zhang, Lanyi Zhu… 展开作者

Ken Gu, Ruoxi Shang, Ruien Jiang, Keying Kuang, Richard-John Lin, Donghe Lyu, Yue Mao, Youran Pan, Teng Wu, Jiaqian Yu, Yikun Zhang, Tianmai M. Zhang, Lanyi Zhu, Mike A. Merrill, Jeffrey Heer, Tim Althoff

AI总结:

本文提出BLADE基准,通过12个数据集和专家基准真值自动评估语言模型智能体在数据驱动科学中的开放式分析决策能力,揭示其分析多样性有所提升但仍未达最优。

摘要:

数据驱动的科学发现需要迭代整合科学领域知识、统计专业知识以及对数据语义的理解,以做出细致入微的分析决策,例如应考虑哪些变量、变换和统计模型。配备规划、记忆和代码执行能力的基于语言模型的智能体有潜力支持数据驱动科学。然而,由于存在多种有效方法、部分正确的步骤以及表达相同决策的不同方式,在此类开放式任务上评估智能体具有挑战性。为应对这些挑战,我们提出了 BLADE,一个用于自动评估智能体对开放式研究问题的多方面处理方法的基准测试。BLADE 包含 12 个数据集和研究问题,这些数据集和研究问题取自现有科学文献,其基准真值由专家数据科学家和研究人员的独立分析收集而来。为自动评估智能体响应,我们开发了相应的计算方法,将分析的不同表示与这些基准真值进行匹配。尽管语言模型拥有相当多的世界知识,但我们的评估表明,它们通常仅限于基础分析。然而,能够与底层数据交互的智能体在其分析决策中表现出改进的但仍非最优的多样性。我们的工作使得对数据驱动科学智能体的评估成为可能,并为研究人员提供了对智能体分析方法的更深入见解。

英文摘要:

Data-driven scientific discovery requires the iterative integration of scientific domain knowledge, statistical expertise, and an understanding of data semantics to make nuanced analytical decisions, e.g., about which variables, transformations, and statistical models to consider. LM-based agents equipped with planning, memory, and code execution capabilities have the potential to support data-driven science. However, evaluating agents on such open-ended tasks is challenging due to multiple valid approaches, partially correct steps, and different ways to express the same decisions. To address these challenges, we present BLADE, a benchmark to automatically evaluate agents' multifaceted approaches to open-ended research questions. BLADE consists of 12 datasets and research questions drawn from existing scientific literature, with ground truth collected from independent analyses by expert data scientists and researchers. To automatically evaluate agent responses, we developed corresponding computational methods to match different representations of analyses to this ground truth. Though language models possess considerable world knowledge, our evaluation shows that they are often limited to basic analyses. However, agents capable of interacting with the underlying data demonstrate improved, but still non-optimal, diversity in their analytical decision making. Our work enables the evaluation of agents for data-driven science and provides researchers deeper insights into agents' analysis approaches.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑