arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2009.13905cs.CL
收藏

通过传递性调整观察者内一致性

Aligning Intraobserver Agreement by Transitivity

The Open University(英国开放大学);School of Computing and Communications(计算与通信学院);Milton Keynes, UK(英国米尔顿凯恩斯)

Jacopo Amidei

摘要:

注释的可重复性和准确性依赖于注释者内部良好的一致性。我们提出了一种新的方法来测量内注释一致性或注释观察者内协议(IA)。所提出的方法是基于传递性,一个措施,已被彻底研究的背景下,理性决策。与注释者IA常用的重测策略相比,及物性测量对重测策略引入的几种类型的偏倚不太敏感。我们提出了一个表示定理的效果,相对判断数据,满足传递性可以映射到一个规模(测量理论方面)。我们还讨论了传递性的进一步应用,作为数据汇集设计的一部分,以解决相对判断的数据汇集的二次复杂性问题。

英文摘要:

Annotation reproducibility and accuracy rely on good consistency within annotators. We propose a novel method for measuring within annotator consistency or annotator Intraobserver Agreement (IA). The proposed approach is based on transitivity, a measure that has been thoroughly studied in the context of rational decision-making. The transitivity measure, in contrast with the commonly used test-retest strategy for annotator IA, is less sensitive to the several types of bias introduced by the test-retest strategy. We present a representation theorem to the effect that relative judgement data that meet transitivity can be mapped to a scale (in terms of measurement theory). We also discuss a further application of transitivity as part of data collection design for addressing the problem of the quadratic complexity of data collection of relative judgements.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑