论专家发现系统的有偏评估
On the Biased Assessment of Expert Finding Systems
AI总结:
本研究针对专家发现系统评估中因自动推荐标注导致的偏差问题,分析偏差来源并提出标注约束方案,以保障评估的公平性与方法比较的有效性。
摘要:
在大型组织中,识别特定主题的专家对于利用分散在各团队和部门的内部知识至关重要。所谓的企业专家检索系统会基于海量的员工异构数据及其工作内容,自动发现并构建员工的专业知识结构。评估这类系统需要全面的真实专家标注,而这类标注难以获取。因此,标注流程通常依赖待验证知识领域的自动推荐。本案例研究分析了这类推荐会如何影响专家发现系统的评估。我们在一个流行基准上证明,经系统验证的标注会导致传统基于术语的检索模型性能被高估,甚至会使其与更新的神经方法的比较失效。我们还为知识领域补充了同义词,以揭示系统对其组成词字面提及的强烈偏差。最后,我们提出了标注流程的约束条件以避免这类有偏评估,并证明这仍能保证标注建议具有高实用性。这些发现应为专家发现的基准构建或选择提供参考,以确保方法比较的有效性。
英文摘要:
In large organisations, identifying experts on a given topic is crucial in leveraging the internal knowledge spread across teams and departments. So-called enterprise expert retrieval systems automatically discover and structure employees' expertise based on the vast amount of heterogeneous data available about them and the work they perform. Evaluating these systems requires comprehensive ground truth expert annotations, which are hard to obtain. Therefore, the annotation process typically relies on automated recommendations of knowledge areas to validate. This case study provides an analysis of how these recommendations can impact the evaluation of expert finding systems. We demonstrate on a popular benchmark that system-validated annotations lead to overestimated performance of traditional term-based retrieval models and even invalidate comparisons with more recent neural methods. We also augment knowledge areas with synonyms to uncover a strong bias towards literal mentions of their constituent words. Finally, we propose constraints to the annotation process to prevent these biased evaluations, and show that this still allows annotation suggestions of high utility. These findings should inform benchmark creation or selection for expert finding, to guarantee meaningful comparison of methods.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。