arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2303.12898cs.CL
收藏

理解医学文本到SQL模型和数据集的泛化

Towards Understanding the Generalization of Medical Text-to-SQL Models and Datasets

The University of Texas at San Antonio(德克萨斯大学圣安东尼奥分校)

Richard Tarbell, Kim-Kwang Raymond Choo, Glenn Dietrich, Anthony Rios

摘要:

电子病历(EMR)存储在关系数据库中。如果用户不熟悉数据库模式或一般数据库基础知识,则访问所需信息可能具有挑战性。因此,研究人员已经探索了文本到SQL生成方法,提供医疗保健专业人员直接存取EMR数据,而不需要数据库专家。然而,目前可用的数据集基本上已经用最先进的模型“解决”,其准确度大于或接近90%。在本文中,我们表明,在解决医疗领域的文本到SQL生成之前,还有很长的路要走。为了证明这一点,我们创建了现有医学文本到SQL数据集MIMICSQL的新分割,以更好地衡量结果模型的泛化能力。我们评估了最先进的语言模型在我们的新分裂显示性能大幅下降,准确率从高达92%下降到28%,从而显示出巨大的改进空间。此外,我们引入了一种新的数据增强方法来提高语言模型的泛化能力。总的来说,本文是在医疗领域开发更强大的文本到SQL模型的第一步。\footnote {数据集和代码将在验收后发布。

英文摘要:

Electronic medical records (EMRs) are stored in relational databases. It can be challenging to access the required information if the user is unfamiliar with the database schema or general database fundamentals. Hence, researchers have explored text-to-SQL generation methods that provide healthcare professionals direct access to EMR data without needing a database expert. However, currently available datasets have been essentially "solved" with state-of-the-art models achieving accuracy greater than or near 90%. In this paper, we show that there is still a long way to go before solving text-to-SQL generation in the medical domain. To show this, we create new splits of the existing medical text-to-SQL dataset MIMICSQL that better measure the generalizability of the resulting models. We evaluate state-of-the-art language models on our new split showing substantial drops in performance with accuracy dropping from up to 92% to 28%, thus showing substantial room for improvement. Moreover, we introduce a novel data augmentation approach to improve the generalizability of the language models. Overall, this paper is the first step towards developing more robust text-to-SQL models in the medical domain.\footnote{The dataset and code will be released upon acceptance.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑