野外数据分析：针对现实世界数据复杂性对大语言模型进行基准测试
Data Analysis in the Wild: Benchmarking Large Language Models Against Real-World Data Complexities
AI总结：
针对大语言模型在数据分析基准测试无法反映现实情况的问题，引入DataGovBench基准测试，含表格问答和表格洞察两个任务，通过实验发现现有模型有性能差距，为推进相关研究提供挑战基准。
摘要：
当前用于评估大语言模型（LLMs）在数据分析方面的基准测试往往无法反映现实世界的情况。它们通常专注于从小表中检索事实，而忽略了大型多表数据集、外部知识整合和探索性洞察发现等挑战。我们引入了DataGovBench，这是一个源自政府开放数据的基准测试，旨在评估LLMs在实际场景中的表现。该基准测试包括两个任务：表格问答，需要解决复杂的可分解问题并生成文本答案或可视化；表格洞察，评估模型通过探索性数据分析生成专家级发现的能力。对最先进的LLMs进行的综合实验表明，在这两个任务上都存在显著的性能差距。这些结果表明，当前基于LLM的系统远不能满足现实世界数据分析的需求。DataGovBench为推进能够回答分析性查询并从数据中发现洞察的LLMs研究提供了具有挑战性的基准测试。代码和示例数据可通过此https链接获取。
英文摘要：
Current benchmarks for evaluating Large Language Models (LLMs) in data analysis often fail to reflect real-world settings. They typically focus on fact retrieval from small tables and overlook the challenges of large multi-tabular datasets, external knowledge integration, and exploratory insight discovery. We introduce DataGovBench, a benchmark derived from governmental open data designed to evaluate LLMs in practical scenarios. The benchmark includes two tasks: Table QA that requires solving complex decomposable questions and producing textual answers or visualizations, and Table Insight that evaluates the ability of models to generate expert-level findings through exploratory data analysis. Comprehensive experiments with state-of-the-art LLMs, both with and without agentic frameworks, reveal significant performance gaps across both tasks. These results suggest that current LLM-based systems remain far from satisfying the demands of real-world data analytics. DataGovBench provides a challenging benchmark for advancing research on LLMs capable of both answering analytical queries and discovering insights from data. Code and sample data are available at https://github.com/SoHasegawa/datagovbench.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
