基于LLM的章节识别器在开源数据上表现优异但在真实世界应用中受挫
LLM-Based Section Identifiers Excel on Open Source but Stumble in Real World Applications
AI总结:
提出利用大型语言模型在零样本和少样本设置下识别EHR章节标题,发现GPT-4在开源数据上优于现有方法,但在真实世界数据集上表现不佳,凸显了需进一步研究和构建更难基准的必要性。
摘要:
电子健康记录(EHR)尽管是医疗从业者的福音,但正变得日益复杂和冗长。在这些冗长的EHR中筛选信息是一项繁重的工作,并成为医患互动中繁琐的一部分。已有多种方法被提出以缓解这一普遍问题,无论是通过摘要还是分节,但过去只有少数方法真正有效。随着自动化方法的兴起,机器学习(ML)在解决识别EHR中相关章节的任务上展现了潜力。然而,大多数ML方法依赖于难以在医疗领域获取的标注数据。另一方面,大型语言模型(LLMs)在自然语言处理(NLP)中取得了令人瞩目的成就,且是以零样本(zero-shot)方式,即无需任何标注数据。为此,我们提出使用LLMs来识别相关的章节标题。我们发现GPT-4能够在零样本和少样本设置下有效解决该任务,并且其分割效果显著优于现有最先进方法。此外,我们还标注了一个难度更大的真实世界数据集,发现GPT-4难以取得良好表现,这表明需要进一步的研究和更具挑战性的基准。
英文摘要:
Electronic health records (EHR) even though a boon for healthcare practitioners, are growing convoluted and longer every day. Sifting around these lengthy EHRs is taxing and becomes a cumbersome part of physician-patient interaction. Several approaches have been proposed to help alleviate this prevalent issue either via summarization or sectioning, however, only a few approaches have truly been helpful in the past. With the rise of automated methods, machine learning (ML) has shown promise in solving the task of identifying relevant sections in EHR. However, most ML methods rely on labeled data which is difficult to get in healthcare. Large language models (LLMs) on the other hand, have performed impressive feats in natural language processing (NLP), that too in a zero-shot manner, i.e. without any labeled data. To that end, we propose using LLMs to identify relevant section headers. We find that GPT-4 can effectively solve the task on both zero and few-shot settings as well as segment dramatically better than state-of-the-art methods. Additionally, we also annotate a much harder real world dataset and find that GPT-4 struggles to perform well, alluding to further research and harder benchmarks.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。