arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2603.17017cs.CLcs.AI
收藏

LLM NL2SQL鲁棒性:传统与智能设置中表面噪声与语言变异的比较

LLM NL2SQL Robustness: Surface Noise vs. Linguistic Variation in Traditional and Agentic Settings

Oracle AI

Lifu Tu, Rongguang Wang, Tao Sheng, Sujjith Ravi, Dan Roth

AI总结:

本文评估了NL2SQL系统在表面噪声和语言变异下的鲁棒性,发现传统和智能设置中模型表现各异,揭示了实现鲁棒NL2SQL系统的挑战。

摘要:

自然语言到SQL(NL2SQL)系统的鲁棒性评估至关重要,因为现实数据库环境是动态、嘈杂且持续变化的,而传统基准评估通常假设静态模式和规范用户输入。本文引入了一个包含约十种扰动类型的鲁棒性评估基准,并在传统和智能设置下进行评估。我们评估了多个最先进的大语言模型(LLMs),包括Grok-4.1、Gemini-3-Pro、Claude-Opus-4.6和GPT-5.2。我们的结果表明,这些模型在几种扰动下表现稳健,但在表面噪声(如字符级破坏)和保持语义但改变词汇或语法形式的语言变异下出现显著性能下降。此外,我们发现表面噪声在传统流程中导致更大的性能下降,而语言变异在智能设置中带来更大挑战。这些发现突显了实现鲁棒NL2SQL系统仍需解决的挑战,特别是在处理语言变异方面。

英文摘要:

Robustness evaluation for Natural Language to SQL (NL2SQL) systems is essential because real-world database environments are dynamic, noisy, and continuously evolving, whereas conventional benchmark evaluations typically assume static schemas and well-formed user inputs. In this work, we introduce a robustness evaluation benchmark containing approximately ten types of perturbations and conduct evaluations under both traditional and agentic settings. We assess multiple state-of-the-art large language models (LLMs), including Grok-4.1, Gemini-3-Pro, Claude-Opus-4.6, and GPT-5.2. Our results show that these models generally maintain strong performance under several perturbations; however, notable performance degradation is observed for surface-level noise (e.g., character-level corruption) and linguistic variation that preserves semantics while altering lexical or syntactic forms. Furthermore, we observe that surface-level noise causes larger performance drops in traditional pipelines, whereas linguistic variation presents greater challenges in agentic settings. These findings highlight the remaining challenges in achieving robust NL2SQL systems, particularly in handling linguistic variability.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑