Recently, there has been significant progress in studying neural networks for translating text descriptions into SQL queries under the zero-shot cross-domain setting. Despite achieving good performance on some public benchmarks, we observe that existing text-to-SQL models do not generalize when facing domain knowledge that does not frequently appear in the training data, which may render the worse prediction performance for unseen domains. In this work, we investigate the robustness of text-to-SQL models when the questions require rarely observed domain knowledge. In particular, we define five types of domain knowledge and introduce Spider-DK (DK is the abbreviation of domain knowledge), a human-curated dataset based on the Spider benchmark for text-to-SQL translation. NL questions in Spider-DK are selected from Spider, and we modify some samples by adding domain knowledge that reflects real-world question paraphrases. We demonstrate that the prediction accuracy dramatically drops on samples that require such domain knowledge, even if the domain knowledge appears in the training set, and the model provides the correct predictions for related training samples.
SUN: Exploring Intrinsic Uncertainties in Text-to-SQL Parsers
Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences(中国科学院深圳先进技术研究院);University of Chinese Academy of Sciences(中国科学院大学);DAMO Academy, Alibaba Group(阿里巴巴集团达摩院)
Exploring the Role of Knowledge Graph-Based RAG in Japanese Medical Question Answering with Small-Scale LLMs
The University of Tokyo(东京大学);NEC Laboratories America(NEC美国实验室);Dokkyo Medical University(多摩医学大学);Graduate School of Frontier Sciences, The University of Tokyo(东京大学前沿科学研究生院)
Exploring Typographic Visual Prompts Injection Threats in Cross-Modality Generation Models
The Hong Kong University of Science and Technology (Guangzhou)(香港科学与技术大学(广州));University of Oxford(牛津大学);Beijing Academy of Artificial Intelligence(北京人工智能研究院);The Hong Kong University of Science and Technology(香港科学与技术大学);Beijing University of Technology(北京工业大学);Tsinghua University(清华大学);City University of Hong Kong(香港城市大学)
Exploring the Potential of Large Language Models for Estimating the Reading Comprehension Question Difficulty
Siebel School of Computing and Data Science, University of Illinois Urbana-Champaign(伊利诺伊大学厄巴纳-香槟分校西贝尔计算与数据科学学院);Arkansas State University(阿肯色州立大学);School of Computer Science, Carnegie Mellon University(卡内基梅隆大学计算机科学学院);Heinz College of Information Systems and Public Policy, Carnegie Mellon University(卡内基梅隆大学海因茨信息系统与公共政策学院);University of Memphis(孟菲斯大学);Institute for Intelligent Systems, University of Memphis(孟菲斯大学智能系统研究所);Department of Psychology, University of Memphis(孟菲斯大学心理学系)
Exploring Iterative Enhancement for Improving Learnersourced Multiple-Choice Question Explanations with Large Language Models
Xtracta(埃克斯特拉克塔公司);School of Computer Science, University of Auckland(奥克兰大学计算机科学学院);School of Computer Science and Engineering, Sun Yat-Sen University(中山大学计算机科学与工程学院);School of Life and Environmental Sciences, University of Sydney(悉尼大学生命与环境科学学院)