arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2401.01943cs.CLcs.AI
收藏

通用嵌入模型在短上下文临床语义搜索中优于专用嵌入模型

Generalist embedding models are better at short-context clinical semantic search than specialized embedding models

Kaduceo;German Heart Center Munich(德国慕尼黑心脏中心)

Jean-Baptiste Excoffier, Tom Roehr, Alexei Figueroa, Jens-Michalis Papaioannou, Keno Bressem, Matthieu Ortala

AI总结:

本研究基于ICD-10-CM代码描述构建数据集进行语义搜索基准测试,发现通用嵌入模型在短上下文临床语义搜索任务中优于专用模型,主要原因是专用模型对输入微小变化更敏感且训练数据多样性不足。

摘要:

在医疗领域,基于大型语言模型(LLMs)的工具和解决方案在各种任务中的日益广泛应用已成为显著趋势。因此,在这个高度关键和敏感的领域中使用它们,引发了关于其鲁棒性(尤其是对输入变化的响应)以及生成输出可靠性的重要问题。本研究通过构建一个基于ICD-10-CM代码描述的文本数据集来解决这些问题,该描述在美国医院广泛使用并包含许多临床术语,以及其易于复现的改写。然后,我们在语义搜索任务中对现有的嵌入模型(通用模型或临床领域专用模型)进行了基准测试,该任务的目标是将改写后的文本与原始描述正确匹配。我们的结果显示,通用模型的表现优于临床模型,这表明现有的临床专用模型对输入中的微小变化更为敏感,这些变化会干扰它们。突出的专用模型问题可能是由于它们未在足够的数据上进行训练,特别是数据集不够多样化,无法实现可靠的全局语言理解,而这对于准确处理医疗文档仍然是必要的。

英文摘要:

The increasing use of tools and solutions based on Large Language Models (LLMs) for various tasks in the medical domain has become a prominent trend. Their use in this highly critical and sensitive domain has thus raised important questions about their robustness, especially in response to variations in input, and the reliability of the generated outputs. This study addresses these questions by constructing a textual dataset based on the ICD-10-CM code descriptions, widely used in US hospitals and containing many clinical terms, and their easily reproducible rephrasing. We then benchmarked existing embedding models, either generalist or specialized in the clinical domain, in a semantic search task where the goal was to correctly match the rephrased text to the original description. Our results showed that generalist models performed better than clinical models, suggesting that existing clinical specialized models are more sensitive to small changes in input that confuse them. The highlighted problem of specialized models may be due to the fact that they have not been trained on sufficient data, and in particular on datasets that are not diverse enough to have a reliable global language understanding, which is still necessary for accurate handling of medical documents.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑