arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2610.17256cs.AI
收藏

受管控的置信分能否判断文本到SQL的答案是否正确?一项针对BIRD的探索性研究及针对Spider的预注册验证

Does a Governed Confidence Score Know When a Text-to-SQL Answer Is Right? An Exploratory Study on BIRD and a Pre-registered Confirmation on Spider

Numberz.ai Inc.(Numberz.ai公司)

Raghu Venkat, Tricha Anjali

AI总结:

本文审计文本到SQL系统的设计置信分,在BIRD上发现其区分答案能力弱,冻结规则后预注册Spider测试,结果显示AUROC提升、正确答案隐藏率下降,验证了调整置信分规则的有效性。

摘要:

对话分析让人们向分布在多个系统中的记录提问并以自然语言获取答案,仅当每个答案可被信任且可解释时才有用。这类生产系统会为其在数据库上生成的每个答案附加一个置信分,用于决定是展示该答案、带警告展示还是隐藏该答案,该分数的权重、上限和阈值均为设计设定。本文对该设计进行审计:在BIRD开发集的1534个问题上,由GPT-5.1采用适配基准的生产提示且每个问题取5个样本进行回答,设计的分数区分正确与错误答案的能力较弱(AUROC为0.624,95%置信区间0.595-0.652),且会隐藏35.7%的正确答案。其几乎所有排序信号来自样本间的一致性,且当一致性与检查清单(对任何无依据主张,包括评论中的主张,均置零)结合时,大部分信号会丢失。我们冻结了一条规则:与展示答案的一致性即为分数,确定性检查作为否决项和上限,并在Spider答案存在前预注册了针对Spider开发集的测试。主要假设成立:AUROC从0.566升至0.725(配对差值+0.159,95%置信区间+0.097至+0.221),正确答案隐藏率从49.0%降至8.8%,PASS准确率为95.7%。在一种匹配策略下,一致性统计量的选择影响不大;提升来自一致性进入分数的方式。附加检查是否有帮助取决于正确性的定义,且它们总会降低合格覆盖率。一项预注册的空结果检查在Spider上未达到其效应量标准。此处的正确性指核心答案;标签由模型判断,一项盲法双人审计(尚未完成)将在修订版中报告。

英文摘要:

Conversational analytics lets people ask questions of records spread across many systems and get answers in plain language; it is useful only if each answer can be trusted and explained. A production system of this kind attaches a confidence score to every answer it writes over a database and uses it to show the answer, show it with a warning, or withhold it. The score's weights, caps and thresholds were set by design. This paper audits that design. On the 1,534 questions of BIRD dev, answered by GPT-5.1 with five samples per question, the designed score separates right from wrong answers weakly (AUROC 0.624) and withholds 35.7% of correct answers. Nearly all of its ranking signal comes from agreement among the samples, and most of it is lost where agreement is combined by a minimum with a checklist that drops to zero on any unsupported claim, including commentary. Before any Spider answer existed we froze and hashed a rule in which agreement with the shown answer is the score, with deterministic vetoes and deterministic and judge-based caps. On Spider dev AUROC rose from 0.566 to 0.725 (+0.159, 95% CI +0.097 to +0.221; database-level CI +0.098 to +0.234), with PASS accuracy 95.7% against a 92.6% base rate. The frozen rule was not the best agreement-based rule on Spider, and the harness and prompts were rebuilt after registration. Added after review: a GPT-5.1 judge asked for P(correct), averaged with agreement, ranked higher in every evaluation (0.837 on Spider), but it grades its own model's answers and its gain shrinks under execution-match labels; one answer per question from a second model family raised agreement's AUROC from 0.749 to 0.800 on Spider, a gain that held under both label definitions. Labels are model-judged; a ten-answer human pilot found no disagreement, and a full audit has not been done.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑