通过传递性调整观察者内一致性
Aligning Intraobserver Agreement by Transitivity
摘要:
注释的可重复性和准确性依赖于注释者内部良好的一致性。我们提出了一种新的方法来测量内注释一致性或注释观察者内协议(IA)。所提出的方法是基于传递性,一个措施,已被彻底研究的背景下,理性决策。与注释者IA常用的重测策略相比,及物性测量对重测策略引入的几种类型的偏倚不太敏感。我们提出了一个表示定理的效果,相对判断数据,满足传递性可以映射到一个规模(测量理论方面)。我们还讨论了传递性的进一步应用,作为数据汇集设计的一部分,以解决相对判断的数据汇集的二次复杂性问题。
英文摘要:
Annotation reproducibility and accuracy rely on good consistency within annotators. We propose a novel method for measuring within annotator consistency or annotator Intraobserver Agreement (IA). The proposed approach is based on transitivity, a measure that has been thoroughly studied in the context of rational decision-making. The transitivity measure, in contrast with the commonly used test-retest strategy for annotator IA, is less sensitive to the several types of bias introduced by the test-retest strategy. We present a representation theorem to the effect that relative judgement data that meet transitivity can be mapped to a scale (in terms of measurement theory). We also discuss a further application of transitivity as part of data collection design for addressing the problem of the quadratic complexity of data collection of relative judgements.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。