arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv cs/0008007cs.CL
收藏

给定层次标记集的标记器评估

Tagger Evaluation Given Hierarchical Tag Sets

I. Dan Melamed, Philip Resnik

摘要:

我们提出了评估人类和自动标签,扩展目前的做法在三个方面的方法。首先,我们将展示如何评估为每个测试实例分配多个标签的标记器,即使它们不分配概率。其次,我们展示了如何适应手动构建的“黄金标准”,通常用于客观评价,即往往有一个以上的正确答案的共同属性。第三,我们将展示如何衡量性能时,一组可能的标签是树结构的IS-A层次结构。为了说明我们的方法可以用来衡量注释者之间的一致性,我们展示了如何计算分层标签集上的kappa系数。

英文摘要:

We present methods for evaluating human and automatic taggers that extend current practice in three ways. First, we show how to evaluate taggers that assign multiple tags to each test instance, even if they do not assign probabilities. Second, we show how to accommodate a common property of manually constructed ``gold standards'' that are typically used for objective evaluation, namely that there is often more than one correct answer. Third, we show how to measure performance when the set of possible tags is tree-structured in an IS-A hierarchy. To illustrate how our methods can be used to measure inter-annotator agreement, we show how to compute the kappa coefficient over hierarchical tag sets.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑