arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2604.28049cs.AI
收藏

不依赖代理的生产环境SQL准确性评估

Agent-Agnostic Evaluation of SQL Accuracy in Production Text-to-SQL Systems

Taslim Jamal Arif, Kuldeep Singh

AI总结:

本文提出STEF框架,通过自然语言输入和生成的SQL进行生产环境文本到SQL系统的评估,无需数据库模式或参考查询,实现持续监控和改进反馈。

摘要:

在生产环境中评估文本到SQL系统面临根本性挑战,现有基准不解决这些问题。当前方法假设可获得真实查询和结构化数据库模式,这在实际部署中很少满足。这种脱节导致生产T2SQL代理在开发时间测试外难以评估,造成无声的质量退化,无反馈机制进行持续改进。我们提出了STEF(模式无关文本到SQL评估框架),一个专为生产环境设计的评估系统,仅需自然语言输入、丰富改写和生成的SQL,无需数据库模式或参考查询。STEF从自然语言和SQL表示中提取语义规范,执行归一化特征对齐,并通过综合指标生成可解释的0到100分准确性评分,涵盖过滤对齐、语义判断和评估者的信心。关键贡献包括:丰富的问题质量验证作为首要评估信号,通过提示模板实现可配置的应用特定规则注入,以及生产稳健的归一化处理GROUP BY容忍度、ORDER BY默认值和LIMIT启发式方法。实证结果表明,STEF能够在不依赖模式的情况下实现持续生产监控和代理改进反馈循环,使结构化查询评估首次在大规模上成为可能。

英文摘要:

Text-to-SQL (T2SQL) evaluation in production environments poses fundamental challenges that existing benchmarks do not address. Current evaluation methodologies whether rule-based SQL matching or schema-dependent semantic parsers assume access to ground-truth queries and structured database schema, constraints that are rarely satisfied in real-world deployments. This disconnect leaves production T2SQL agents largely unevaluated beyond developer-time testing, creating silent quality degradation with no feedback mechanism for continuous improvement. We present STEF (Schema-agnostic Text-to-SQL Evaluation Framework), a production-native evaluation system that operates exclusively on natural language inputs the user question, an enriched reformulation, and the generated SQL without requiring database schema or reference queries. STEF extracts semantic specifications from both natural language and SQL representations, performs normalized feature alignment, and produces an interpretable 0 to 100 accuracy score via a composite metric that encompasses filter alignment, semantic verdict, and confidence of the evaluator. Key contributions include: enriched question quality validation as a first-class evaluation signal, configurable application-specific rule injection via prompt templating, and production-robust normalization handling GROUP BY tolerance, ORDER BY defaults, and LIMIT heuristics. Empirical results demonstrate that STEF enables continuous production monitoring and agent improvement feedback loops without schema dependency, making structured query evaluation viable at scale for the first time.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑