arXiv 2511.041532025-11-06cs.CLcs.AIcs.DBcs.MA

BAPPA：面向自动化文本到SQL生成的智能体、计划与流水线基准评测

BAPPA: Benchmarking Agents, Plans, and Pipelines for Automated Text-to-SQL Generation

Center for Computational & Data Sciences, Independent University, Bangladesh（计算与数据科学中心，独立大学，孟加拉国）；University of Massachusetts Amherst（马萨诸塞大学阿默斯特分校）；University of Illinois Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）

Fahim Ahmed, Md Mubtasim Ahasan, Jahir Sadik Monon, Muntasir Wahed, M Ashraful Amin, A K M Mahbubur Rahman, Amin Ahsan Ali

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

本文提出BAPPA基准，系统评测三种多智能体LLM流水线在文本到SQL任务上的表现，发现多智能体讨论可提升小模型准确率，其中推理器-编码器流水线效果最佳，将Gemma 3 27B IT执行准确率提升至56.4%。

摘要：

文本到SQL系统提供了一种自然语言接口，使非专业人士也能访问存储在数据库中的信息。然而，现有的大型语言模型（LLM）在面对大规模数据库模式和复杂推理时，难以根据自然语言指令生成SQL。此前的工作往往侧重于使用旗舰模型构建复杂且不太实用的流水线，而较小、高效的模型则被忽视。在本工作中，我们探索了三种多智能体LLM流水线，并在一系列从小到大的开源模型上进行了系统性的性能基准评测：（1）多智能体讨论流水线，其中智能体迭代地批评和改进SQL查询，并由一个裁判智能体综合出最终答案；（2）规划器-编码器流水线，其中一个思考型规划器模型生成逐步的SQL生成计划，再由一个编码器综合出查询；（3）编码器-聚合器流水线，其中多个编码器独立生成SQL查询，然后由一个推理智能体选择最佳查询。在Bird-Bench Mini-Dev数据集上的实验表明，多智能体讨论能够提升小模型的性能，Qwen2.5-7b-Instruct经过三轮讨论后执行准确率最高提升了10.6%。在这些流水线中，LLM推理器-编码器流水线取得了最佳结果，DeepSeek-R1-32B和QwQ-32B规划器将Gemma 3 27B IT的准确率从52.4%提升至最高分56.4%。代码可在https://github.com/treeDweller98/bappa-sql获取。

英文摘要：

Text-to-SQL systems provide a natural language interface that can enable even laymen to access information stored in databases. However, existing Large Language Models (LLM) struggle with SQL generation from natural instructions due to large schema sizes and complex reasoning. Prior work often focuses on complex, somewhat impractical pipelines using flagship models, while smaller, efficient models remain overlooked. In this work, we explore three multi-agent LLM pipelines, with systematic performance benchmarking across a range of small to large open-source models: (1) Multi-agent discussion pipeline, where agents iteratively critique and refine SQL queries, and a judge synthesizes the final answer; (2) Planner-Coder pipeline, where a thinking model planner generates stepwise SQL generation plans and a coder synthesizes queries; and (3) Coder-Aggregator pipeline, where multiple coders independently generate SQL queries, and a reasoning agent selects the best query. Experiments on the Bird-Bench Mini-Dev set reveal that Multi-Agent discussion can improve small model performance, with up to 10.6% increase in Execution Accuracy for Qwen2.5-7b-Instruct seen after three rounds of discussion. Among the pipelines, the LLM Reasoner-Coder pipeline yields the best results, with DeepSeek-R1-32B and QwQ-32B planners boosting Gemma 3 27B IT accuracy from 52.4% to the highest score of 56.4%. Codes are available at https://github.com/treeDweller98/bappa-sql.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。