Existing Text-to-SQL generators require the entire schema to be encoded with the user text. This is expensive or impractical for large databases with tens of thousands of columns. Standard dense retrieval techniques are inadequate for schema subsetting of a large structured database, where the correct semantics of retrieval demands that we rank sets of schema elements rather than individual elements. In response, we propose a two-stage process for effective coverage during retrieval. First, we instruct an LLM to hallucinate a minimal DB schema deemed adequate to answer the query. We use the hallucinated schema to retrieve a subset of the actual schema, by composing the results from multiple dense retrievals. Remarkably, hallucination $\unicode{x2013}$ generally considered a nuisance $\unicode{x2013}$ turns out to be actually useful as a bridging mechanism. Since no existing benchmarks exist for schema subsetting on large databases, we introduce three benchmarks. Two semi-synthetic datasets are derived from the union of schemas in two well-known datasets, SPIDER and BIRD, resulting in 4502 and 798 schema elements respectively. A real-life benchmark called SocialDB is sourced from an actual large data warehouse comprising 17844 schema elements. We show that our method1 leads to significantly higher recall than SOTA retrieval-based augmentation methods.
EH-Benchmark Ophthalmic Hallucination Benchmark and Agent-Driven Top-Down Traceable Reasoning Workflow
Institute of High Performance Computing, Agency for Science, Technology and Research (A*STAR)(高性能计算研究所,科技研究局(A*STAR));Centre for Innovation and Precision Eye Health(创新与精准眼健康中心);Department of Ophthalmology, NUHS Tower Block, Level 7, 1E Kent Ridge Road, Singapore, 119228(眼科部,NUHS塔楼7层,1E Kent Ridge Road,新加坡,119228);Singapore Eye Research Institute, Singapore National Eye Centre, 20 College Road, Singapore, 169856(新加坡眼研究 institute,新加坡国家眼科中心,20 College Road,新加坡,169856)
Enhancing Text-to-SQL Capabilities of Large Language Models via Domain Database Knowledge Injection
School of Cyber Science Engineering, Huazhong University of Science and Technology(华中科技大学网络空间科学与工程学院);Wuhan AI Research Foundation Model Research Center(武汉人工智能研究基金会大模型研究中心);Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所);School of Artificial Intelligence, University of Chinese Academy of Sciences(中国科学院大学人工智能学院);Peng Cheng Laboratory(鹏城实验室)