LLMJudge: 用于相关性判断的大语言模型
LLMJudge: LLMs for Relevance Judgments
AI总结：
SIGIR 2024的LLMJudge挑战赛探索用LLMs生成相关性判断以替代人工标注，将研究模型效果、提示词优化等问题并发布数据支撑相关研究。
摘要：
LLMJudge挑战赛是作为SIGIR 2024举办的LLM4Eval研讨会的一部分组织开展的。测试集对于评估信息检索（IR）系统至关重要，搜索系统的评估与调优很大程度上依赖相关性标签，这类标签用于表明某一文档对特定搜索请求和用户是否有用。但大规模收集相关性判断成本高昂、耗费资源，因此常规实验依赖第三方标注人员，而这类人员的标注结果未必始终准确。LLMJudge挑战赛旨在探索一种替代方案：使用大语言模型（LLMs）生成相关性判断。已有研究表明LLMs能够为搜索系统生成可靠的相关性判断，但目前仍有诸多问题尚不明确：哪些LLMs的准确率能与人类标注者相当、哪种提示词效果最佳、经过微调的开源LLMs与GPT-4这类闭源LLMs相比表现如何、合成生成的数据是否存在偏差，以及数据泄露是否会影响生成标签的质量。本次挑战赛将对这些问题展开研究，收集到的数据将打包发布，以支持信息检索与搜索领域的自动相关性判断研究。
英文摘要：
The LLMJudge challenge is organized as part of the LLM4Eval workshop at SIGIR 2024. Test collections are essential for evaluating information retrieval (IR) systems. The evaluation and tuning of a search system is largely based on relevance labels, which indicate whether a document is useful for a specific search and user. However, collecting relevance judgments on a large scale is costly and resource-intensive. Consequently, typical experiments rely on third-party labelers who may not always produce accurate annotations. The LLMJudge challenge aims to explore an alternative approach by using LLMs to generate relevance judgments. Recent studies have shown that LLMs can generate reliable relevance judgments for search systems. However, it remains unclear which LLMs can match the accuracy of human labelers, which prompts are most effective, how fine-tuned open-source LLMs compare to closed-source LLMs like GPT-4, whether there are biases in synthetically generated data, and if data leakage affects the quality of generated labels. This challenge will investigate these questions, and the collected data will be released as a package to support automatic relevance judgment research in information retrieval and search.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
