PICO跨度标注一致性研究
A Study on Agreement in PICO Span Annotations
AI总结:
本文研究专家与非专家在PICO文本跨度标注中的一致性,发现边界差异大但大致区域共识高,主张联合使用标准与宽松一致性指标。
摘要:
在循证医学中,医学文献的相关性由预定义的相关性条件决定。这些条件基于PICO要素定义,即患者(Patient)、干预(Intervention)、对照(Comparator)和结局(Outcome)。因此,医学文献中的PICO标注对于自动相关文献过滤至关重要。然而,为PICO要素界定文本跨度的边界并非易事。本文研究了由多个人类标注者(包括专家和非专家)完成的PICO标注的一致性。一致性通过一种标准跨度一致性(即同时匹配文本跨度的标签和边界)以及两种宽松跨度一致性(即匹配标签但不保证匹配跨度边界)进行评估。基于分析,我们报告了两点观察:(i)单个人类标注者对PICO跨度标注的边界非常多样;(ii)尽管跨度边界存在分歧,但标注者对跨度标注的大致区域存在广泛共识。我们的结果表明,仅采用标准一致性可能会低估PICO跨度的一致性,而同时采用标准一致性和宽松一致性更适合PICO跨度评估。
英文摘要:
In evidence-based medicine, relevance of medical literature is determined by predefined relevance conditions. The conditions are defined based on PICO elements, namely, Patient, Intervention, Comparator, and Outcome. Hence, PICO annotations in medical literature are essential for automatic relevant document filtering. However, defining boundaries of text spans for PICO elements is not straightforward. In this paper, we study the agreement of PICO annotations made by multiple human annotators, including both experts and non-experts. Agreements are estimated by a standard span agreement (i.e., matching both labels and boundaries of text spans), and two types of relaxed span agreement (i.e., matching labels without guaranteeing matching boundaries of spans). Based on the analysis, we report two observations: (i) Boundaries of PICO span annotations by individual human annotators are very diverse. (ii) Despite the disagreement in span boundaries, general areas of the span annotations are broadly agreed by annotators. Our results suggest that applying a standard agreement alone may undermine the agreement of PICO spans, and adopting both a standard and a relaxed agreements is more suitable for PICO span evaluation.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。