基于跨度级与文档级特征分类的荷兰语超声心动图非结构化报告诊断提取
Diagnosis extraction from unstructured Dutch echocardiogram reports using span- and document-level characteristic classification
AI总结：
本研究针对荷兰语非结构化超声心动图报告，验证了自动提取诊断的可行性，对比了跨度级与文档级分类方法，推荐了最优模型并给出小数据场景的替代方案。
摘要：
临床机器学习研究与AI驱动的临床决策支持模型依赖于临床准确的标签。在临床专家协助下手动提取这些标签通常耗时且成本高昂。本研究测试了从非结构化荷兰语超声心动图报告中自动进行跨度级与文档级诊断提取的可行性。我们纳入了来自荷兰大型大学医院UMCU的115692份非结构化超声心动图报告。我们随机选择了一个子集，针对11种常见描述的心脏特征的出现情况与严重程度进行了人工标注。我们开发并测试了多种跨度级与文档级的自动标注技术，采用加权F1值、宏F1值、精确率和召回率进行性能评估。我们对比了跨度标注方法与文档标注方法的性能，其中文档标注方法包括直接文档分类器和依赖跨度分类结果的间接文档分类器。SpanCategorizer（跨度分类器）和MedRoBERTa.nl模型分别在所有跨度分类器和文档分类器中表现最优。不同特征的加权F1值存在差异，SpanCategorizer的加权F1值在0.60到0.93之间，MedRoBERTa.nl的加权F1值在0.96到0.98之间。直接文档分类的效果优于基于跨度分类器的间接文档分类。SetFit仅使用10%的训练数据就取得了具有竞争力的文档分类性能。使用简化标签集可获得近乎完美的文档分类结果。我们推荐使用已发布的SpanCategorizer和MedRoBERTa.nl模型从荷兰语超声心动图报告中进行跨度级与文档级诊断提取。对于训练数据有限的场景，SetFit可能是文档分类的一种有前景的替代方案。
英文摘要：
Clinical machine learning research and AI driven clinical decision support models rely on clinically accurate labels. Manually extracting these labels with the help of clinical specialists is often time-consuming and expensive. This study tests the feasibility of automatic span- and document-level diagnosis extraction from unstructured Dutch echocardiogram reports. We included 115,692 unstructured echocardiogram reports from the UMCU a large university hospital in the Netherlands. A randomly selected subset was manually annotated for the occurrence and severity of eleven commonly described cardiac characteristics. We developed and tested several automatic labelling techniques at both span and document levels, using weighted and macro F1-score, precision, and recall for performance evaluation. We compared the performance of span labelling against document labelling methods, which included both direct document classifiers and indirect document classifiers that rely on span classification results. The SpanCategorizer and MedRoBERTa$.$nl models outperformed all other span and document classifiers, respectively. The weighted F1-score varied between characteristics, ranging from 0.60 to 0.93 in SpanCategorizer and 0.96 to 0.98 in MedRoBERTa$.$nl. Direct document classification was superior to indirect document classification using span classifiers. SetFit achieved competitive document classification performance using only 10% of the training data. Utilizing a reduced label set yielded near-perfect document classification results. We recommend using our published SpanCategorizer and MedRoBERTa$.$nl models for span- and document-level diagnosis extraction from Dutch echocardiography reports. For settings with limited training data, SetFit may be a promising alternative for document classification.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
