理解、检测和修复基于上下文学习的真实世界文本到SQL错误
Understanding, Detecting, and Repairing Real-World In-Context-Learning-Based Text-to-SQL Errors
AI总结:
本研究首次全面调查基于上下文学习的文本到SQL错误,总结27种错误类型,并提出MapleDoctor框架,相比现有方法修复率提高13.8%,误修复极少,延迟降低67.4%。
摘要:
大型语言模型(LLMs)已被用于文本到SQL任务,利用其上下文学习(ICL)能力将自然语言问题转换为SQL查询。然而,这种技术面临正确性问题。在本文中,我们首次对基于ICL的文本到SQL错误进行了全面研究。我们的研究涵盖了四种代表性的ICL技术、五种基本修复方法、两个基准测试和两种LLM设置。我们发现文本到SQL错误普遍存在,并总结了7个类别的27种错误类型。我们还发现,现有的修复尝试在正确性提升方面有限,同时具有高计算开销和许多误修复。基于这些发现,我们提出了MapleDoctor,一种新颖的文本到SQL错误检测和修复框架。评估表明,MapleDoctor优于现有解决方案,修复了13.8%更多的查询,误修复数量可忽略不计,并减少了67.4%的修复延迟。该工件可在GitHub上公开获取。
英文摘要:
Large language models (LLMs) have been adopted for text-to-SQL tasks, utilizing their in-context learning (ICL) capability to translate natural language questions into SQL queries. However, such a technique faces correctness problems. In this paper, we conduct the first comprehensive study of text-to-SQL errors of ICL-based techniques. Our study covers four representative ICL-based techniques, five basic repairing methods, two benchmarks, and two LLM settings. We find that text-to-SQL errors are widespread and summarize 27 error types of 7 categories. We also find that existing repairing attempts have limited correctness improvement while having high computational overhead and many mis-repairs. Based on these findings, we propose MapleDoctor, a novel text-to-SQL error detection and repairing framework. The evaluation demonstrates that MapleDoctor outperforms existing solutions by repairing 13.8% more queries with a negligible number of mis-repairs and reducing 67.4% repair latency. The artifact is publicly available at GitHub.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。