自然语言处理、信息检索（IR）、计算机视觉等技术的快速发展给评估这些系统的性能带来了重大挑战。主要挑战之一是人类标注数据的稀缺，这阻碍了对这些系统公平且准确的评估。在本研究中，我们专门关注评估具有稀疏标签的IR系统，借鉴了近期关于计算机视觉任务评估的研究，并从Fréchet Inception Distance（FID）在评估文本到图像生成系统中的成功应用中获得启发。我们提出利用Fréchet Distance来度量相关判定项分布与检索结果分布之间的距离。我们在MS MARCO V1数据集和TREC Deep Learning Tracks查询集上的实验结果表明，Fréchet Distance作为评估IR系统的指标是有效的，特别是在只有少量标签可用的情况下。该方法有助于推动生成式IR系统评估等真实场景中评估方法的进步。
英文摘要：
The rapid advancement of natural language processing, information retrieval (IR), computer vision, and other technologies has presented significant challenges in evaluating the performance of these systems. One of the main challenges is the scarcity of human-labeled data, which hinders the fair and accurate assessment of these systems. In this work, we specifically focus on evaluating IR systems with sparse labels, borrowing from recent research on evaluating computer vision tasks. taking inspiration from the success of using Fréchet Inception Distance (FID) in assessing text-to-image generation systems. We propose leveraging the Fréchet Distance to measure the distance between the distributions of relevant judged items and retrieved results. Our experimental results on MS MARCO V1 dataset and TREC Deep Learning Tracks query sets demonstrate the effectiveness of the Fréchet Distance as a metric for evaluating IR systems, particularly in settings where a few labels are available. This approach contributes to the advancement of evaluation methodologies in real-world scenarios such as the assessment of generative IR systems.
A Generative Framework for Personalized Sticker Retrieval
State Key Laboratory of AI Safety, Institute of Computing Technology, CAS（中国科学院计算技术研究所人工智能安全国家重点实验室）；University of Chinese Academy of Sciences（中国科学院大学）；WeChat Search Application Department, Tencent Inc.（腾讯公司微信搜索应用部）
WeKnow-RAG: An Adaptive Approach for Retrieval-Augmented Generation Integrating Web Search and Knowledge Graphs
Ping An Bank Co., Ltd.（平安银行股份有限公司）；Tsinghua University（清华大学）；Central South University（中南大学）；Shanghai Yiwo Information Technology Co., Ltd.（上海易沃信息技术有限公司）；Kwangwoon University（光云大学）
