LogicCat：面向复杂推理的思维链文本到SQL基准测试
LogicCat: A Chain-of-Thought Text-to-SQL Benchmark for Complex Reasoning
AI总结：
提出首个针对复杂推理和思维链解析的Text-to-SQL基准数据集LogicCat，涵盖物理、算术、常识和假设推理场景，通过4038个问题与12114条思维链步骤显著提升任务难度，现有模型执行准确率最高仅33.20%。
摘要：
文本到SQL是自然语言处理中的关键任务，旨在将自然语言问题转化为准确且可执行的SQL查询。在现实场景中，这些推理任务通常伴随复杂的数学计算、领域知识和假设推理场景。然而，现有大规模文本到SQL数据集通常聚焦于业务逻辑和任务逻辑，忽略了垂直领域知识、复杂数学推理和假设推理等关键因素，而这些因素对于真实反映实际应用中的推理需求并完成数据查询与分析至关重要。为弥补这一空白，我们引入了LogicCat，这是首个专门为复杂推理和思维链解析设计的文本到SQL基准数据集，涵盖物理、算术、常识和假设推理场景。LogicCat包含4038个英文问题，配有12114条详细的思维链推理步骤，跨越45个不同领域的数据库，在复杂性上显著超越现有数据集。实验结果表明，LogicCat将当前最先进模型的任务难度大幅提升至最高33.20%的执行准确率，表明该任务仍然极具挑战性。LogicCat的进步代表了向开发适用于真实企业数据分析和自主查询生成的系统迈出的关键一步。我们已在https://github.com/Ffunkytao/LogicCat发布了数据集代码。
英文摘要：
Text-to-SQL is a critical task in natural language processing that aims to transform natural language questions into accurate and executable SQL queries. In real-world scenarios, these reasoning tasks are often accompanied by complex mathematical computations, domain knowledge, and hypothetical reasoning scenarios. However, existing large-scale Text-to-SQL datasets typically focus on business logic and task logic, neglecting critical factors such as vertical domain knowledge, complex mathematical reasoning, and hypothetical reasoning, which are essential for realistically reflecting the reasoning demands in practical applications and completing data querying and analysis. To bridge this gap, we introduce LogicCat, the first Text-to-SQL benchmark dataset specifically designed for complex reasoning and chain-of-thought parsing, encompassing physics, arithmetic, commonsense, and hypothetical reasoning scenarios. LogicCat comprises 4,038 English questions paired 12,114 detailed chain-of-thought reasoning steps, spanning 45 databases across diverse domains, significantly surpassing existing datasets in complexity. Experimental results demonstrate that LogicCat substantially increases the task difficulty for current state-of-the-art models to at most 33.20% execution accuracy, indicating that this task remains exceptionally challenging. The advancement of LogicCat represents a crucial step toward developing systems suitable for real-world enterprise data analysis and autonomous query generation. We have released our dataset code at https://github.com/Ffunkytao/LogicCat.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
