arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2211.09455cs.CL
收藏

咨询检查表:规范医疗记录生成的人工评估

Consultation Checklists: Standardising the Human Evaluation of Medical Note Generation

Babylon(巴比伦);University of Aberdeen(阿伯丁大学);ADAPT Research Centre, Dublin City University(都柏林城市大学ADAPT研究中心)

Aleksandar Savkov, Francesco Moramarco, Alex Papadopoulos Korfiatis, Mark Perera, Anya Belz, Ehud Reiter

摘要:

由于输出质量的许多方面固有的主观性质,评估自动生成的文本通常是困难的。这种困难在自动会诊记录生成中由于医学专家之间关于哪些患者陈述应包括在所生成的记录中以及关于它们各自在得出诊断中的重要性的不同意见而变得复杂。以前的笔记生成系统的真实世界的评估看到专家评估员之间的实质性分歧。在本文中,我们提出了一个协议,旨在提高客观性接地评估咨询清单,这是在初步步骤中创建,然后作为一个共同的参考点,在质量评估。在使用该协议的第一次评估研究中,我们观察到注释者之间的协议水平良好;此外,与使用原始人类注释相比,使用研究中产生的咨询检查表作为ROUGE或BERTScore等自动度量的参考,提高了它们与人类判断的相关性。

英文摘要:

Evaluating automatically generated text is generally hard due to the inherently subjective nature of many aspects of the output quality. This difficulty is compounded in automatic consultation note generation by differing opinions between medical experts both about which patient statements should be included in generated notes and about their respective importance in arriving at a diagnosis. Previous real-world evaluations of note-generation systems saw substantial disagreement between expert evaluators. In this paper we propose a protocol that aims to increase objectivity by grounding evaluations in Consultation Checklists, which are created in a preliminary step and then used as a common point of reference during quality assessment. We observed good levels of inter-annotator agreement in a first evaluation study using the protocol; further, using Consultation Checklists produced in the study as reference for automatic metrics such as ROUGE or BERTScore improves their correlation with human judgements compared to using the original human note.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑