Recent discussions on alternative facts, fake news, and post truth politics have motivated research on creating technologies that allow people not only to access information, but also to assess the credibility of the information presented to them by information retrieval systems. Whereas technology is in place for filtering information according to relevance and/or credibility, no single measure currently exists for evaluating the accuracy or precision (and more generally effectiveness) of both the relevance and the credibility of retrieved results. One obvious way of doing so is to measure relevance and credibility effectiveness separately, and then consolidate the two measures into one. There at least two problems with such an approach: (I) it is not certain that the same criteria are applied to the evaluation of both relevance and credibility (and applying different criteria introduces bias to the evaluation); (II) many more and richer measures exist for assessing relevance effectiveness than for assessing credibility effectiveness (hence risking further bias).
Motivated by the above, we present two novel types of evaluation measures that are designed to measure the effectiveness of both relevance and credibility in ranked lists of retrieval results. Experimental evaluation on a small human-annotated dataset (that we make freely available to the research community) shows that our measures are expressive and intuitive in their interpretation.
SemEval-2013 Task 4: Free Paraphrases of Noun Compounds
Radboud University Nijmegen(拉德堡德大学);Universidade de Lisboa(里斯本大学);QCRI, Qatar Foundation(卡塔尔基金会卡塔尔计算研究所);University of Ottawa(渥太华大学);Polish Academy of Sciences(波兰科学院);University of Southern California(南加州大学);University of Cambridge(剑桥大学);University College Dublin(都柏林大学学院)
AxIoU: An Axiomatically Justified Measure for Video Moment Retrieval
Cyberagent, Inc.(赛博艾坚特公司);Waseda University(早稻田大学);Osaka University(大阪大学);Tampere University(坦佩雷大学);University of Oulu(奥卢大学)
评价措施对研究方向有着至关重要的影响。因此,对于常规方法不太适用的新应用,开发适当和可靠的评估方法至关重要。视频矩检索(VMR)就是这样一种应用,并且当前的实践是使用R@ $K,θ$来评估VMR系统。然而,这一措施有两个缺点。首先,它是排名不敏感的:它通过将列表视为集合而忽略排名前$K$的列表中成功定位的时刻的排名位置。其次,它使用阈值$θ$将每个检索到的视频时刻的交集(IoU)二值化,从而忽略排名时刻的细粒度定位质量。我们提出了一种替代措施,用于评估VMR,称为平均最大IoU(AxIoU),这是免费的上述两个问题。我们表明,AxIoU满足两个重要的公理VMR评估,即\textbf{Invariance against Redundant Moments}和\textbf{Monotonicity with respect to the Best Moment},也R@ $K,θ$满足第一个公理。我们还从经验上研究了AxIoU与R@ $K,θ$的一致性,以及它在测试数据和人类注释的时间边界变化方面的稳定性。
Rethinking Literature Search Evaluation: Deep Research Helps, and Human Citation Lists Are Not a Ground Truth
Mila – Quebec AI Institute(魁北克AI研究所);HEC Montréal(蒙特利尔HEC商学院);ServiceNow Research(ServiceNow研究);Canada CIFAR AI Chair(加拿大CIFAR人工智能主席);Université de Montréal(蒙特利尔大学);Polytechnique Montréal(蒙特利尔理工学院)