基于大语言模型的文本到SQL、文本到Python和文本到函数的再提升——及其在交通领域的实际应用
Reboost Large Language Model-based Text-to-SQL, Text-to-Python, and Text-to-Function -- with Real Applications in Traffic Domain
AI总结:
本文针对业务数据集中Text-to-SQL性能下降问题,提出查询重写与SQL增强的通用提示方法,将执行准确率从21.05提升至65.79,并进一步比较Text-to-Python与Text-to-Function方案。
摘要:
此前最先进(SOTA)方法在Spider数据集上取得了显著的执行准确率,Spider是Text-to-SQL领域规模最大、多样性最强的数据集之一。然而,我们在复现业务数据集时观察到性能显著下降。我们考察了数据集复杂性和问题意图清晰度方面的差异,并评估了这些差异如何影响提示方法的性能。随后,我们开发了一种更具适应性和更通用的提示方法,主要涉及查询重写和SQL增强,分别将模糊信息转化为精确信息,并通过引入执行反馈和数据库内容中的查询结果来增强SQL本身。为防止信息缺口,我们在提示中将列的注释、值类型和值样本作为数据库描述的一部分。我们在大语言模型(LLMs)上的实验表明,该方法在业务数据集上性能显著提升,并证明了其巨大潜力。在业务数据集的执行准确率方面,SOTA方法得分为21.05,而我们的方法得分为65.79。因此,即使使用能力较弱的预训练语言模型,我们的方法也取得了显著的性能提升。最后,我们还探索了Text-to-Python和Text-to-Function选项,并深入分析了它们之间的优缺点,为社区提供了有价值的见解。
英文摘要:
The previous state-of-the-art (SOTA) method achieved a remarkable execution accuracy on the Spider dataset, which is one of the largest and most diverse datasets in the Text-to-SQL domain. However, during our reproduction of the business dataset, we observed a significant drop in performance. We examined the differences in dataset complexity, as well as the clarity of questions' intentions, and assessed how those differences could impact the performance of prompting methods. Subsequently, We develop a more adaptable and more general prompting method, involving mainly query rewriting and SQL boosting, which respectively transform vague information into exact and precise information and enhance the SQL itself by incorporating execution feedback and the query results from the database content. In order to prevent information gaps, we include the comments, value types, and value samples for columns as part of the database description in the prompt. Our experiments with Large Language Models (LLMs) illustrate the significant performance improvement on the business dataset and prove the substantial potential of our method. In terms of execution accuracy on the business dataset, the SOTA method scored 21.05, while our approach scored 65.79. As a result, our approach achieved a notable performance improvement even when using a less capable pre-trained language model. Last but not least, we also explore the Text-to-Python and Text-to-Function options, and we deeply analyze the pros and cons among them, offering valuable insights to the community.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。