arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 1708.07157cs.IR
收藏

排行榜的相关性和可信性评价指标

Evaluation Measures for Relevance and Credibility in Ranked Lists

Christina Lioma, Jakob Grue Simonsen, Birger Larsen

摘要:

最近关于替代事实、假新闻和后真相政治的讨论激发了人们对创造技术的研究,这些技术不仅允许人们访问信息,而且还允许人们评估信息检索系统提供给他们的信息的可信度。虽然已经有根据相关性和/或可信度过滤信息的技术,但目前还没有一种单一的措施来评估检索结果的相关性和可信度的准确性或精确性(以及更普遍的有效性)。这样做的一个明显方法是分别衡量相关性和可信性有效性,然后将这两个衡量标准合并为一个。这种办法至少有两个问题:(一)不能肯定是否对相关性和可信性的评价适用相同的标准(采用不同的标准会给评价带来偏见);(二)评估相关性有效性的措施比评估可信性有效性的措施多得多,内容也更丰富(因此有进一步偏见的风险)。出于上述动机,我们提出了两种新类型的评估措施,旨在衡量检索结果的排名列表中的相关性和可信度的有效性。对一个小型人类注释数据集(我们免费提供给研究社区)的实验评估表明,我们的措施在解释上是富有表现力和直观的。

英文摘要:

Recent discussions on alternative facts, fake news, and post truth politics have motivated research on creating technologies that allow people not only to access information, but also to assess the credibility of the information presented to them by information retrieval systems. Whereas technology is in place for filtering information according to relevance and/or credibility, no single measure currently exists for evaluating the accuracy or precision (and more generally effectiveness) of both the relevance and the credibility of retrieved results. One obvious way of doing so is to measure relevance and credibility effectiveness separately, and then consolidate the two measures into one. There at least two problems with such an approach: (I) it is not certain that the same criteria are applied to the evaluation of both relevance and credibility (and applying different criteria introduces bias to the evaluation); (II) many more and richer measures exist for assessing relevance effectiveness than for assessing credibility effectiveness (hence risking further bias). Motivated by the above, we present two novel types of evaluation measures that are designed to measure the effectiveness of both relevance and credibility in ranked lists of retrieval results. Experimental evaluation on a small human-annotated dataset (that we make freely available to the research community) shows that our measures are expressive and intuitive in their interpretation.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑