增强金融系统设计的Text-to-SQL翻译
Enhancing Text-to-SQL Translation for Financial System Design
AI总结:
本文研究金融系统设计中的Text-to-SQL任务,基于LLMs进行性能与输入优化基准测试,提出两种衡量SQL查询相似度的新指标,并证明基于树的编辑距离可高效可靠地评估生成查询质量。
摘要:
Text-to-SQL,即将自然语言问题翻译为SQL查询的任务,是各种业务流程的一部分。其自动化是一项新兴挑战,将赋能软件从业者使用自然语言与关系数据库无缝交互,从而弥合业务需求与软件能力之间的差距。在本文中,我们考虑了在各种NLP任务中达到最先进水平的大型语言模型(LLMs)。具体而言,我们对Text-to-SQL性能、评估方法以及输入优化(例如提示工程)进行了基准测试。根据我们得出的经验观察结果,我们提出了两个旨在充分衡量SQL查询之间相似度的新颖指标。总体而言,我们与社区分享了多项发现,特别是在Text-to-SQL任务中如何选择合适的LLM。我们进一步证明,基于树的编辑距离构成了评估生成的SQL查询与基准真相之间相似度的可靠指标,可用于对Text2SQL方法进行基准测试。该指标非常重要,因为它使研究人员无需执行计算成本高昂的实验,例如像先前工作那样执行生成的查询。我们的工作实现了金融领域的用例,因此有助于推动Text2SQL系统的发展及其在该领域的实际应用。
英文摘要:
Text-to-SQL, the task of translating natural language questions into SQL queries, is part of various business processes. Its automation, which is an emerging challenge, will empower software practitioners to seamlessly interact with relational databases using natural language, thereby bridging the gap between business needs and software capabilities. In this paper, we consider Large Language Models (LLMs), which have achieved state of the art for various NLP tasks. Specifically, we benchmark Text-to-SQL performance, the evaluation methodologies, as well as input optimization (e.g., prompting). In light of the empirical observations that we have made, we propose two novel metrics that were designed to adequately measure the similarity between SQL queries. Overall, we share with the community various findings, notably on how to select the right LLM on Text-to-SQL tasks. We further demonstrate that a tree-based edit distance constitutes a reliable metric for assessing the similarity between generated SQL queries and the oracle for benchmarking Text2SQL approaches. This metric is important as it relieves researchers from the need to perform computationally expensive experiments such as executing generated queries as done in prior works. Our work implements financial domain use cases and, therefore contributes to the advancement of Text2SQL systems and their practical adoption in this domain.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。