Text-to-SQL任务支持任何人通过自然语言从SQL数据库中检索信息。尽管该任务已取得长足进展，但两大核心评估指标——执行准确率（Execution Accuracy, EXE）和精确集匹配准确率（Exact Set Matching Accuracy, ESM）存在固有局限性，可能导致性能表征失实。具体而言，ESM的刚性匹配机制会忽略语义正确但表述风格不同的查询，而EXE会忽略那些能输出正确结果的结构错误，从而高估正确率。在评估无需微调的大语言模型（LLM）类方法的输出时，这些缺陷的问题尤为突出——因为与经过微调的同类方法相比，无微调LLM的输出在风格和结构上差异更大。为此，我们提出了一种新指标——增强树匹配（Enhanced Tree Matching, ETM），它通过同时结合句法和语义要素比对查询，从而缓解上述问题。通过对9个基于LLM的模型进行评估，我们发现EXE和ESM的假阳性率和假阴性率可分别高达23.0%和28.9%，而ETM将这两个比率分别降至0.3%和2.7%。我们将ETM脚本开源，为社区提供一种更稳健、更可靠的Text-to-SQL评估方法。
英文摘要：
The task of Text-to-SQL enables anyone to retrieve information from SQL databases using natural language. While this task has made substantial progress, the two primary evaluation metrics - Execution Accuracy (EXE) and Exact Set Matching Accuracy (ESM) - suffer from inherent limitations that can misrepresent performance. Specifically, ESM's rigid matching overlooks semantically correct but stylistically different queries, whereas EXE can overestimate correctness by ignoring structural errors that yield correct outputs. These shortcomings become especially problematic when assessing outputs from large language model (LLM)-based approaches without fine-tuning, which vary more in style and structure compared to their fine-tuned counterparts. Thus, we introduce a new metric, Enhanced Tree Matching (ETM), which mitigates these issues by comparing queries using both syntactic and semantic elements. Through evaluating nine LLM-based models, we show that EXE and ESM can produce false positive and negative rates as high as 23.0% and 28.9%, while ETM reduces these rates to 0.3% and 2.7%, respectively. We release our ETM script as open source, offering the community a more robust and reliable approach to evaluating Text-to-SQL.
SQLFixAgent: Towards Semantic-Accurate Text-to-SQL Parsing via Consistency-Enhanced Multi-Agent Collaboration
School of Computer Science & Technology, Soochow University（苏州大学计算机科学与技术学院）；School of Information, Renmin University of China（中国人民大学信息学院）；International College (Suzhou Research Institute), Renmin University of China（中国人民大学国际学院（苏州研究院））；iFLYTEK Research (Suzhou)（科大讯飞研究院（苏州））
Cooperative SQL Generation for Segmented Databases By Using Multi-functional LLM Agents
School of Computer Science, Northwestern Polytechnical University（西北工业大学计算机学院）；Big Data Shortage and Management, MIIT（工业和信息化部大数据短缺与管理局）；School of Computing, National University of Singapore（新加坡国立大学计算机学院）；School of Computing and Information Systems, Singapore Management University（新加坡管理大学计算与信息系统学院）
ProSPy: A Profiling-Driven SQL-Python Agentic Framework for Enterprise Text-to-SQL
State Key Lab of CAD&CG（计算机辅助设计与图形学国家重点实验室）；School of Software Technology（软件技术学院）；Tencent TEG（腾讯科技集团）；School of Mathematical Sciences, Peking University（北京大学数学科学学院）；Zhejiang University（浙江大学）
$R^3$: "This is My SQL, Are You With Me?" A Consensus-Based Multi-Agent System for Text-to-SQL Tasks
School of Engineering, Westlake University（西湖大学工程学院）；University of Michigan（密歇根大学）；Carnegie Mellon University（卡内基梅隆大学）；School of Mathematical Science, Shanghai Jiao Tong University（上海交通大学数学科学学院）
BAPPA: Benchmarking Agents, Plans, and Pipelines for Automated Text-to-SQL Generation
Center for Computational & Data Sciences, Independent University, Bangladesh（计算与数据科学中心，独立大学，孟加拉国）；University of Massachusetts Amherst（马萨诸塞大学阿默斯特分校）；University of Illinois Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）
VTS-LLM: Domain-Adaptive LLM Agent for Enhancing Awareness in Vessel Traffic Services through Natural Language
Institute of High Performance Computing, Agency for Science, Technology and Research (A*STAR IHPC)（高性能计算研究所，科技研究局（A*STAR IHPC））；National University of Singapore（国立新加坡大学）；Shanghai University（上海大学）
Evaluating and Enhancing LLMs for Multi-turn Text-to-SQL with Multiple Question Types
School of Computer Science, Harbin University of Science and Technology（哈尔滨理工大学计算机科学学院）；Faculty of Computing, Harbin Institute of Technology（哈尔滨工业大学计算学部）
BiomedSQL: Text-to-SQL for Scientific Reasoning on Biomedical Knowledge Bases
Center for Alzheimer’s and Related Dementias, NIA, NIH（阿尔茨海默病及相关痴呆症研究中心，国家老龄化研究所，国立卫生研究院）；DataTecnica LLC（DataTecnica公司）；Johns Hopkins University（约翰霍普金斯大学）；Laboratory of Neurogenetics, NIA, NIH（神经遗传学实验室，国家老龄化研究所，国立卫生研究院）
A Multi-agent Text2SQL Framework using Small Language Models and Execution Feedback
Griffith University (Australia)（格里菲斯大学）；VinUniversity (Vietnam)（文大学）；Humboldt-Universitat zu Berlin (Germany)（柏林洪堡大学）；The University of Queensland, Australia (2025)（昆士兰大学）
