MT-Teql：使用变形测试评估和增强文本到SQL模型的一致性
MT-Teql: Evaluating and Augmenting Consistency of Text-to-SQL Models with Metamorphic Testing
摘要：
Text-to-SQL是从人类话语生成SQL查询的任务。然而，由于自然语言的变异性，两个语义等价的话语在词汇层面上可能会出现不同的表现。同样，用户偏好（例如，范式的选择）在表达概念上相同的模式时可能导致表结构的显著变化。设想文本到SQL模型的一般困难，以保持预测的一致性对语言和模式的变化，我们提出了MT-Teql，一个基于变形测试的框架，系统地评估和增强的一致性TEXT到SQL模型。受软件变形测试原理的启发，MT-Teql提供了一个与模型无关的框架，该框架实现了一组全面的变形关系（MR），以进行对话语和模式的语义保持转换。当原始输入和转换后的输入导致不同的SQL查询时，可能会暴露模型不一致性。此外，我们利用转换后的输入来重新训练模型，以进一步提高模型的鲁棒性。我们的实验表明，我们的框架暴露了SOTA模型的数千个预测错误，并按数量级丰富了现有数据集，消除了超过40%的不一致性错误，而不影响标准准确性。
英文摘要：
Text-to-SQL is a task to generate SQL queries from human utterances. However, due to the variation of natural language, two semantically equivalent utterances may appear differently in the lexical level. Likewise, user preferences (e.g., the choice of normal forms) can lead to dramatic changes in table structures when expressing conceptually identical schemas. Envisioning the general difficulty for text-to-SQL models to preserve prediction consistency against linguistic and schema variations, we propose MT-Teql, a Metamorphic Testing-based framework for systematically evaluating and augmenting the consistency of TExt-to-SQL models. Inspired by the principles of software metamorphic testing, MT-Teql delivers a model-agnostic framework which implements a comprehensive set of metamorphic relations (MRs) to conduct semantics-preserving transformations toward utterances and schemas. Model Inconsistency can be exposed when the original and transformed inputs induce different SQL queries. In addition, we leverage the transformed inputs to retrain models for further model robustness boost. Our experiments show that our framework exposes thousands of prediction errors from SOTA models and enriches existing datasets by order of magnitude, eliminating over 40% inconsistency errors without compromising standard accuracy.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
