TrustSQL:基于惩罚评分的文本到SQL可靠性基准测试
TrustSQL: Benchmarking Text-to-SQL Reliability with Penalty-Based Scoring
AI总结:
TrustSQL基准通过惩罚评分评估文本到SQL模型的可靠性,要求模型对可行问题正确生成SQL、对不可行问题弃权(不执行),并比较流水线与统一方法,为更安全部署提供新视角。
摘要:
文本到SQL(Text-to-SQL)使用户能够使用自然语言与数据库进行交互,简化了信息的检索与综合。尽管大语言模型(LLMs)在将自然语言问题翻译为SQL查询方面取得了显著成功,但由于两个主要挑战,广泛部署仍然受限。首先,文本到SQL模型的有效使用取决于用户对模型能力的理解——即模型能够正确回答的问题范围。其次,缺乏弃权(不执行)机制可能导致错误的SQL生成未被察觉,从而削弱对模型输出的信任。为了实现更广泛的部署,关键在于在模型设计中应对这些挑战,并加强模型评估以建立对模型输出的信任。为此,我们引入了TrustSQL,这是一个新颖且全面的基准测试,旨在评估文本到SQL的可靠性——其定义为模型正确处理任何类型输入问题的能力:对可行问题生成正确的SQL查询,并对不可行问题(例如,由于模式不兼容或超出SQL的功能限制)弃权(不执行)生成。我们使用一种新颖的基于惩罚的评分指标,通过两种建模方法评估现有方法:(1)基于流水线的方法,将SQL生成器与不可行问题检测器和SQL错误检测器相结合以实现弃权(不执行);(2)统一方法,使用单一模型完成整个任务。我们的实验结果表明,在严厉惩罚下获得高分需要付出显著努力,并为开发更安全部署的文本到SQL模型提供了新的视角。TrustSQL可在https://github.com/glee4810/TrustSQL获取。
英文摘要:
Text-to-SQL enables users to interact with databases using natural language, simplifying the retrieval and synthesis of information. Despite the remarkable success of large language models (LLMs) in translating natural language questions into SQL queries, widespread deployment remains limited due to two primary challenges. First, the effective use of text-to-SQL models depends on users' understanding of the model's capabilities-the scope of questions the model can correctly answer. Second, the absence of abstention mechanisms can lead to incorrect SQL generation going unnoticed, thereby undermining trust in the model's output. To enable wider deployment, it is crucial to address these challenges in model design and enhance model evaluation to build trust in the model's output. To this end, we introduce TrustSQL, a novel comprehensive benchmark designed to evaluate text-to-SQL reliability-defined as a model's ability to correctly handle any type of input question by generating correct SQL queries for feasible questions and abstaining from generating infeasible ones (e.g., due to schema incompatibility or functionalities beyond SQL). We evaluate existing methods using a novel penalty-based scoring metric with two modeling approaches: (1) pipeline-based methods combining SQL generators with infeasible question detectors and SQL error detectors for abstention; and (2) unified methods using a single model for the entire task. Our experimental results reveal that achieving high scores under severe penalties requires significant effort and provide a new perspective on developing text-to-SQL models for safer deployment. TrustSQL is available at https://github.com/glee4810/TrustSQL.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。