arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2510.26495cs.DBcs.CL
收藏

重新思考Text-to-SQL:面向真实数据库探索的动态多轮SQL交互

Rethinking Text-to-SQL: Dynamic Multi-turn SQL Interaction for Real-world Database Exploration

University of Chinese Academy of Sciences(中国科学院大学);Peking University(北京大学);Tsinghua University(清华大学)

Linzhuang Sun, Tianyu Guo, Hao Liang, Yuying Li, Qifeng Cai, Jingxuan Wei, Bihui Yu, Wentao Zhang, Bin Cui

AI总结:

针对现有Text-to-SQL系统无法适配真实场景多轮动态交互的问题,本文构建自动化生成的DySQL-Bench评测基准与多轮交互评估框架,实验证明该基准难度较高,现有顶尖模型表现仍有较大提升空间。

摘要:

Text-to-SQL领域的最新进展已在静态单轮任务上取得优异表现,这类任务中模型可根据自然语言问题生成SQL查询。但此类系统无法适配真实交互场景:用户意图会动态演变,查询需经过多轮迭代优化。在金融、商业分析等应用中,用户会根据中间结果反复调整查询约束或维度。为评估该动态能力,我们推出DySQL-Bench,这一基准用于评测模型在用户交互动态演变场景下的性能。不同于此前人工构建的数据集,DySQL-Bench通过任务合成与验证的两阶段自动化流水线搭建:从原始数据库表提取的结构化树表示引导基于大语言模型的任务生成,后续经过面向交互场景的过滤与专家校验,人工评估确认合成数据正确率达100%。我们进一步提出多轮评估框架,模拟由大语言模型扮演的用户、待测模型与可执行数据库间的真实交互,模型需随用户意图变化调整推理过程与SQL生成逻辑。DySQL-Bench覆盖BIRD与Spider 2数据库中的13个领域,共包含1072个任务。即使GPT-4o在该基准上也仅取得58.34%的整体准确率,Pass@5指标仅为23.81%,凸显了该基准的难度。所有代码与数据已发布于https://github.com/Aurora-slz/Real-World-SQL-Bench。

英文摘要:

Recent advances in Text-to-SQL have achieved strong results in static, single-turn tasks, where models generate SQL queries from natural language questions. However, these systems fall short in real-world interactive scenarios, where user intents evolve and queries must be refined over multiple turns. In applications such as finance and business analytics, users iteratively adjust query constraints or dimensions based on intermediate results. To evaluate such dynamic capabilities, we introduce DySQL-Bench, a benchmark assessing model performance under evolving user interactions. Unlike previous manually curated datasets, DySQL-Bench is built through an automated two-stage pipeline of task synthesis and verification. Structured tree representations derived from raw database tables guide LLM-based task generation, followed by interaction-oriented filtering and expert validation. Human evaluation confirms 100% correctness of the synthesized data. We further propose a multi-turn evaluation framework simulating realistic interactions among an LLM-simulated user, the model under test, and an executable database. The model must adapt its reasoning and SQL generation as user intents change. DySQL-Bench covers 13 domains across BIRD and Spider 2 databases, totaling 1,072 tasks. Even GPT-4o attains only 58.34% overall accuracy and 23.81% on the Pass@5 metric, underscoring the benchmark's difficulty. All code and data are released at https://github.com/Aurora-slz/Real-World-SQL-Bench .

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑