高召回率,小数据:实时法律搜索系统中系统内评估的挑战
High Recall, Small Data: The Challenges of Within-System Evaluation in a Live Legal Search System
AI总结:
本文基于实时法律搜索系统日志和用户研究,揭示了传统排序评估方法在法律IR中因系统特性与用户数据有限而失效的问题,并提出未来应转向基于成本的评估模型。
摘要:
本文阐述了法律信息检索(IR)中常见排序评估方法所面临的一些挑战。我们通过一个实时法律搜索系统的日志数据和两项用户研究来展示这些挑战。我们概述了法律IR的各个方面,以及这些方面对常见评估方法预期挑战的影响:基于显式和隐式反馈的测试集、用户调查和A/B测试。接下来,我们利用一个实时商业法律搜索引擎的数据来说明常见评估方法的挑战。我们特别关注用于监测单个IR系统随时间对文档排序进行(持续)更改的有效性的方法。我们展示了法律IR系统的特征组合与有限的用户数据如何导致挑战,从而使所讨论的常见评估方法变得次优。因此,在未来的工作中,我们将专注于不太常见的评估方法,例如基于成本的评估模型。
英文摘要:
This paper illustrates some challenges of common ranking evaluation methods for legal information retrieval (IR). We show these challenges with log data from a live legal search system and two user studies. We provide an overview of aspects of legal IR, and the implications of these aspects for the expected challenges of common evaluation methods: test collections based on explicit and implicit feedback, user surveys, and A/B testing. Next, we illustrate the challenges of common evaluation methods using data from a live, commercial, legal search engine. We specifically focus on methods for monitoring the effectiveness of (continuous) changes to document ranking by a single IR system over time. We show how the combination of characteristics in legal IR systems and limited user data can lead to challenges that cause the common evaluation methods discussed to be sub-optimal. In our future work we will therefore focus on less common evaluation methods, such as cost-based evaluation models.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。