DiscoveryBench：迈向使用大型语言模型的数据驱动发现
DiscoveryBench: Towards Data-Driven Discovery with Large Language Models
AI总结：
提出包含264个真实与903个合成任务的 DiscoveryBench 基准，形式化数据驱动发现流程并评估LLM框架，结果显示最佳系统仅得25%。
摘要：
使用大型语言模型（LLM）进行代码生成、函数调用和数据分析的快速进展，能否帮助纯粹从一组给定数据集中自动化完成假设的搜索与验证？为评估这一问题，我们提出 DiscoveryBench，这是首个将数据驱动发现的多步骤过程形式化的综合基准。该基准旨在系统评估当前模型在发现任务中的能力，并为改进这些能力提供有用资源。我们的基准包含跨社会学和工程学等6个不同领域收集的264项任务，通过从已发表论文中人工推导发现工作流，以近似研究人员面临的真实挑战；每项任务由一个数据集、其元数据和一个自然语言形式的发现目标定义。我们另外提供903项合成任务，以跨任务复杂度开展受控评估。此外，我们对数据驱动发现的结构化形式化支持基于方面的评估，从而为不同失败模式提供有用洞见。我们在 DiscoveryBench 上使用开源和闭源 LLM 评估了若干流行的基于 LLM 的推理框架作为基线，发现即使最佳系统也仅获得25%的分数。因此，我们的基准阐明了自主数据驱动发现中的挑战，并作为社区取得进展的宝贵资源。
英文摘要：
Can the rapid advances in code generation, function calling, and data analysis using large language models (LLMs) help automate the search and verification of hypotheses purely from a set of provided datasets? To evaluate this question, we present DiscoveryBench, the first comprehensive benchmark that formalizes the multi-step process of data-driven discovery. The benchmark is designed to systematically assess current model capabilities in discovery tasks and provide a useful resource for improving them. Our benchmark contains 264 tasks collected across 6 diverse domains, such as sociology and engineering, by manually deriving discovery workflows from published papers to approximate the real-world challenges faced by researchers, where each task is defined by a dataset, its metadata, and a discovery goal in natural language. We additionally provide 903 synthetic tasks to conduct controlled evaluations across task complexity. Furthermore, our structured formalism of data-driven discovery enables a facet-based evaluation that provides useful insights into different failure modes. We evaluate several popular LLM-based reasoning frameworks using both open and closed LLMs as baselines on DiscoveryBench and find that even the best system scores only 25%. Our benchmark, thus, illustrates the challenges in autonomous data-driven discovery and serves as a valuable resource for the community to make progress.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
