重新评估Netflix奖-人类的不确定性及其对可靠性的影响
Re-Evaluating the Netflix Prize - Human Uncertainty and its Impact on Reliability
摘要:
在本文中,我们研究的统计可靠性比较评估领域内的推荐系统的可靠性和人的不确定性。从一个对照实验中,我们得到的见解,用户提供不同的评级相同的项目时,反复询问。用户评分的这种波动性证明了使用概率密度而不是单个评分分数的假设是合理的。因此,公知的准确性度量(例如,MAE、MSE、RMSE)产生从所有评级密度的卷积中出现的密度本身。当两个不同的系统产生具有显著交集的不同RMSE分布时,则每个可能的排名都存在错误的概率。作为一个应用程序,我们检查Netflix奖可能的排名错误。我们能够表明,所有的顶级排名或多或少受到高概率的错误,有些排名可能被认为是由纯粹的机会,而不是系统质量。
英文摘要:
In this paper, we examine the statistical soundness of comparative assessments within the field of recommender systems in terms of reliability and human uncertainty. From a controlled experiment, we get the insight that users provide different ratings on same items when repeatedly asked. This volatility of user ratings justifies the assumption of using probability densities instead of single rating scores. As a consequence, the well-known accuracy metrics (e.g. MAE, MSE, RMSE) yield a density themselves that emerges from convolution of all rating densities. When two different systems produce different RMSE distributions with significant intersection, then there exists a probability of error for each possible ranking. As an application, we examine possible ranking errors of the Netflix Prize. We are able to show that all top rankings are more or less subject to high probabilities of error and that some rankings may be deemed to be caused by mere chance rather than system quality.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。