Does a Governed Confidence Score Know When a Text-to-SQL Answer Is Right? An Exploratory Study on BIRD and a Pre-registered Confirmation on Spider
Numberz.ai Inc.(Numberz.ai公司)
Raghu Venkat, Tricha Anjali
备注:
26 pages, 4 figures, 12 tables. v2: revised after external review; adds a correctness judge, a second-model check, database-level intervals and a human label pilot, and corrects the reproducibility statement. Code, frozen scorer (branch prereg), analysis scripts and data: https://github.com/numberz-ai/numberz-ies-tdd-h-calibration (release v2.0)
Conversational analytics lets people ask questions of records spread across many systems and get answers in plain language; it is useful only if each answer can be trusted and explained. A production system of this kind attaches a confidence score to every answer it writes over a database and uses it to show the answer, show it with a warning, or withhold it. The score's weights, caps and thresholds were set by design. This paper audits that design. On the 1,534 questions of BIRD dev, answered by GPT-5.1 with five samples per question, the designed score separates right from wrong answers weakly (AUROC 0.624) and withholds 35.7% of correct answers. Nearly all of its ranking signal comes from agreement among the samples, and most of it is lost where agreement is combined by a minimum with a checklist that drops to zero on any unsupported claim, including commentary. Before any Spider answer existed we froze and hashed a rule in which agreement with the shown answer is the score, with deterministic vetoes and deterministic and judge-based caps. On Spider dev AUROC rose from 0.566 to 0.725 (+0.159, 95% CI +0.097 to +0.221; database-level CI +0.098 to +0.234), with PASS accuracy 95.7% against a 92.6% base rate. The frozen rule was not the best agreement-based rule on Spider, and the harness and prompts were rebuilt after registration. Added after review: a GPT-5.1 judge asked for P(correct), averaged with agreement, ranked higher in every evaluation (0.837 on Spider), but it grades its own model's answers and its gain shrinks under execution-match labels; one answer per question from a second model family raised agreement's AUROC from 0.749 to 0.800 on Spider, a gain that held under both label definitions. Labels are model-judged; a ten-answer human pilot found no disagreement, and a full audit has not been done.
Spider 2.0: Evaluating Language Models on Real-World Enterprise Text-to-SQL Workflows
University of Hong Kong(香港大学);Salesforce Research(Salesforce 研究院);Sea AI Lab(Sea AI 实验室);Google Deepmind(谷歌DeepMind);Google Cloud AI Research(谷歌云 AI 研究院);University of Waterloo(滑铁卢大学)
The Ohio State University(俄亥俄州立大学);Microsoft Research(微软研究院)
学习捕获文本表对齐对于像文本到SQL这样的任务是必不可少的。模型需要正确识别对列和值的自然语言引用,并将它们置于给定的数据库模式中。在本文中,我们提出了一种新的弱监督结构接地预训练框架(StruG)的文本到SQL,可以有效地学习捕获基于并行文本表语料库的文本表对齐。我们确定了一组新的预测任务:列接地,值接地和列值映射,并利用它们来预训练文本表编码器。此外,为了在更现实的文本表对齐设置下评估不同的方法,我们基于Spider dev set创建了一个新的评估集Spider-Realistic,其中删除了明确提到的列名,并采用八个现有的文本到SQL数据集进行跨数据库评估。在所有设置中,BERT-LARGE带来了显着的改进。与现有的预训练方法(如GRAPPA)相比,BLOG在Spider上实现了类似的性能,并且在更真实的集合上优于所有基线。Spider-Realistic数据集可在https://doi.org/10.5281/zenodo.5205322上获得。
LLM-Based SQL Generation: Prompting, Self-Refinement, and Adaptive Weighted Majority Voting
Institute of Information Management(信息管理研究所);National Yang Ming Chiao Tung University(阳明交通大学);R. B. Annis School of Engineering(R. B. Annis工程学院);University of Indianapolis(印第安纳大学)
$R^3$: "This is My SQL, Are You With Me?" A Consensus-Based Multi-Agent System for Text-to-SQL Tasks
School of Engineering, Westlake University(西湖大学工程学院);University of Michigan(密歇根大学);Carnegie Mellon University(卡内基梅隆大学);School of Mathematical Science, Shanghai Jiao Tong University(上海交通大学数学科学学院)
EviLink: Multi-Path Schema Linking with Uncertainty-Guided Evidence Acquisition for Large-Scale Text-to-SQL
School of Software Technology, Zhejiang University(浙江大学软件学院);State Key Lab of CAD&CG, Zhejiang University(浙江大学CAD&CG国家重点实验室);Tencent TEG(腾讯TEG);School of Mathematical Sciences, Peking University(北京大学数学科学学院)
Database Context Compression for Text-to-SQL on Real-World Large Databases
National Engineering Research Center of Software Engineering, Peking University(软件工程国家工程研究中心,北京大学);School of Computer Science, Peking University(北京大学计算机学院);Peking University Information Technology Institute, Tianjin Binhai(天津滨海大学信息技术研究所,北京大学)
MTIR-SQL: Multi-turn Tool-Integrated Reasoning Reinforcement Learning for Text-to-SQL
Meituan(美团);Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所);School of Artificial Intelligence, University of Chinese Academy of Sciences(中国科学院大学人工智能学院);Beijing Institute of Technology(北京理工大学);Shanghai Jiao Tong University(上海交通大学);Peking University(北京大学)
SQL-o1: Automatic Text-to-SQL Program Synthesis via Self-Reward Heuristic Search
School of Computer Science, Beijing University of Posts and Telecommunications, China(北京邮电大学计算机科学学院);College of Computing and Data Science, Nanyang Technological University, Singapore(南洋理工大学计算机与数据科学学院);State Key Laboratory of Networking and Switching Technology, Beijing University of Posts and Telecommunications, China(北京邮电大学网络与交换技术国家重点实验室);College of Computer Science, Sichuan University, Chengdu, China(四川大学计算机学院)