PremiseSQL：评估Text-to-SQL系统中的前提批判与弃权（不执行）
PremiseSQL: Evaluating Premise Critique and Abstention in Text-to-SQL Systems
AI总结：
本研究针对Text-to-SQL系统默认信任查询前提的问题，提出PremiseSQL基准与评估框架，发现强化批判提示可提升缺陷查询的检测与弃权（不执行）能力，但会增加误判并抑制原始查询的SQL生成。
摘要：
大型语言模型（LLMs）的近期进展大幅提升了Text-to-SQL的性能，但现有的NL2SQL系统大多假设用户查询是有效且可回答的。因此，即便查询存在有缺陷的前提，模型仍常生成可执行的SQL。我们在NL2SQL中引入前提批判能力，即识别有缺陷的查询并弃权（不执行）以生成误导性SQL的能力。为评估该能力，我们提出PremiseSQL基准，包含1800个跨四类前提失败的有缺陷NL2SQL查询及其对应的1800个原始查询。我们还设计了一个评估框架，用于测量三种提示设置下内在及提示诱导的前提感知行为。对前沿LLMs的实验表明，当前模型默认高度信任前提，极少批判有缺陷的前提；更强的面向批判的提示可提升对有缺陷查询的检测与弃权（不执行）能力，但也会增加错误的无效判断，并抑制对原始查询的SQL生成。代码可在该https URL获取。
英文摘要：
Recent advances in large language models (LLMs) have substantially improved Text-to-SQL performance, yet existing NL2SQL systems largely assume that user queries are valid and answerable. As a result, models often generate executable SQL even for queries with flawed premises. We introduce premise critique ability in NL2SQL, namely the ability to recognize flawed queries and abstain from generating misleading SQL. To evaluate this capability, we propose PremiseSQL, a benchmark containing 1,800 flawed NL2SQL queries across four categories of premise failures, along with their 1,800 original queries. We further design an evaluation framework to measure intrinsic and prompt-induced premise-aware behavior across three prompting settings. Experiments on frontier LLMs show that current models are strongly premise-trusting by default and rarely critique flawed premises. Stronger critique-oriented prompting improves detection and abstention on flawed queries, but also increases false invalid judgments and suppresses SQL generation on originals. The code is available at https://github.com/vegeet/PremiseSQL.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
