生产环境文本到SQL流水线中LLM作为评判者的失败审计与修复
Auditing and Repairing LLM-as-Judge Failures in a Production Text-to-SQL Pipeline
浏览论文内容
中文总结 AI 辅助
本文审计生产环境文本到SQL流水线中的LLM评判者，发现其与人类一致性极低，归因于GRADE-HALLUCINATION机制，并证明自托管Qwen模型以极低成本达到与Claude相当的一致性，且强评判者集成可进一步提升。
中文摘要 AI 辅助
生产环境中的文本到SQL流水线通常以LLM作为评判者收尾，但其与人类标注者的一致性从未被实际测量过。当我们检查自己的流水线时，部署的gpt-4o-mini评判者在分歧增强集上与两位作者的金标准仅达到Cohen's kappa = 0.04，在均匀随机抽查集上为0.42，并在增强集中过度标记了77.1%的人类忠实案例。其大部分过度标记可追溯至我们称之为GRADE-HALLUCINATION的单一机制。自托管的Qwen3.6-27B替代模型（kappa = 0.72）与Claude Opus 4.7（kappa = 0.71）处于同一水平；在n = 96时头对头比较统计功效不足，但对于部署决策这几乎无关紧要，因为Qwen每次调用成本约为其1/300。集成并不能免费带来帮助。将弱评判者与强评判者配对会降低一致性，而三个强评判者在一致路由下达到kappa = 0.79，自动覆盖率为89.7%。跨领域应用时，相同的审计方案在我们的标注协议下标记了BIRD金融数据集中25.5%的专家撰写的金标准SQL为候选金标准问题。代码和预注册见此https URL。
英文摘要
Production text-to-SQL pipelines often end with an LLM-as-judge whose agreement with human annotators has never actually been measured. When we checked ours, the deployed gpt-4o-mini judge agreed with two-author gold at only Cohen's kappa = 0.04 on a disagreement-enriched set and 0.42 on a uniform-random spot-check, over-flagging 77.1% of the human-FAITHFUL cases in the enriched set. Most of its over-flags trace back to a single mechanism we call GRADE-HALLUCINATION. A self-hosted Qwen3.6-27B replacement (kappa = 0.72) lands in the same range as Claude Opus 4.7 (kappa = 0.71); the head-to-head is underpowered at n = 96, but for the deployment decision that hardly matters, since Qwen costs roughly 1/300 as much per call. Ensembling does not help for free. Pairing the weak judge with a stronger one degrades agreement, whereas three strong judges under unanimity routing reach kappa = 0.79 at 89.7% auto-coverage. Applied out-of-domain, the same audit recipe flags 25.5% of BIRD-financial's expert-authored gold SQLs as candidate gold-SQL issues under our annotation protocol. Code and pre-registration are at https://github.com/JamesL404/synca-audit.
发表机构
- Santa Clara University（圣克拉拉大学）
机构由 AI 辅助整理，请以论文原文为准。
