arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新，休息一下吧，周末愉快~~
arXiv 2409.19014cs.CLcs.IRcs.LG
收藏

FLEX：面向可靠 Text-to-SQL 基准的专家级无误执行指标

FLEX: Expert-level False-Less EXecution Metric for Reliable Text-to-SQL Benchmark

Yanolja（야놀자（Yanolja））；Ajou University（明知大学）

Heegyu Kim, Taeyang Jeon, Seunghwan Choi, Seungtaek Choi, Hyunsouk Cho

备注：
preprint, under review
阅读 PDF
arXiv 页面 ↗HTML 原文 ↗

AI总结：

本文提出基于 LLM 模拟专家判断的 FLEX 指标，以解决 EX 在 Text-to-SQL 评估中的假阳性和假阴性问题，并显著提升人类一致性及基准排名可靠性。

摘要：

Text-to-SQL 系统已成为各行业中将自然语言转换为 SQL 查询的关键工具，使非技术用户能够执行复杂的数据操作。随着这些系统日益复杂，对准确评估方法的需求也不断增加。然而，最常用的评估指标 Execution Accuracy（EX）仍存在大量假阳性和假阴性。因此，本文提出 FLEX（False-Less EXecution），这是一种利用大型语言模型（LLM）模拟人类专家对 SQL 查询进行专家级评估的新颖 Text-to-SQL 系统评估方法。我们的指标通过全面的上下文和精细的标准，将与人类专家的一致性从 Cohen's kappa 62 提升至 87.04。大量实验得出若干关键发现：（1）模型性能平均提高超过 2.6 分，显著影响 Spider 和 BIRD 基准上的排名；（2）EX 对模型的低估主要源于标注质量问题；（3）模型在特别困难问题上的性能往往被高估。本工作有助于更准确、细致地评估 Text-to-SQL 系统，并可能重塑我们对该领域最先进性能的理解。

英文摘要：

Text-to-SQL systems have become crucial for translating natural language into SQL queries in various industries, enabling non-technical users to perform complex data operations. The need for accurate evaluation methods has increased as these systems have grown more sophisticated. However, the Execution Accuracy (EX), the most prevalent evaluation metric, still shows many false positives and negatives. Thus, this paper introduces FLEX (False-Less EXecution), a novel approach to evaluating text-to-SQL systems using large language models (LLMs) to emulate human expert-level evaluation of SQL queries. Our metric improves agreement with human experts (from 62 to 87.04 in Cohen's kappa) with comprehensive context and sophisticated criteria. Our extensive experiments yield several key insights: (1) Models' performance increases by over 2.6 points on average, substantially affecting rankings on Spider and BIRD benchmarks; (2) The underestimation of models in EX primarily stems from annotation quality issues; and (3) Model performance on particularly challenging questions tends to be overestimated. This work contributes to a more accurate and nuanced evaluation of text-to-SQL systems, potentially reshaping our understanding of state-of-the-art performance in this field.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。

↑