arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2204.00447cs.CL
收藏

会诊记录生成中的人工评估与自动标注的相关性

Human Evaluation and Correlation with Automatic Metrics in Consultation Note Generation

Babylon(巴比伦(医疗公司));University of Aberdeen(阿伯丁大学)

Francesco Moramarco, Alex Papadopoulos Korfiatis, Mark Perera, Damir Juric, Jack Flann, Ehud Reiter, Anya Belz, Aleksandar Savkov

摘要:

近年来,机器学习模型在生成临床咨询笔记方面已经迅速变得更好;然而,关于如何正确评估生成的咨询笔记以了解它们可能对使用它们的临床医生和患者的临床安全产生的影响的工作很少。为了解决这个问题,我们提出了一项广泛的咨询笔记人类评估研究,其中5名临床医生(i)听取57次模拟咨询,(ii)编写自己的笔记,(iii)后编辑许多自动生成的笔记,以及(iv)提取所有错误,包括定量和定性错误。然后,我们进行了相关性研究与18个自动质量指标和人类的判断。我们发现,一个简单的,基于字符的Levenshtein距离度量执行标准杆,如果不是更好地比常见的基于模型的指标,如BertScore。我们所有的发现和注释都是开源的。

英文摘要:

In recent years, machine learning models have rapidly become better at generating clinical consultation notes; yet, there is little work on how to properly evaluate the generated consultation notes to understand the impact they may have on both the clinician using them and the patient's clinical safety. To address this we present an extensive human evaluation study of consultation notes where 5 clinicians (i) listen to 57 mock consultations, (ii) write their own notes, (iii) post-edit a number of automatically generated notes, and (iv) extract all the errors, both quantitative and qualitative. We then carry out a correlation study with 18 automatic quality metrics and the human judgements. We find that a simple, character-based Levenshtein distance metric performs on par if not better than common model-based metrics like BertScore. All our findings and annotations are open-sourced.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑