不要使用LLM进行相关性判断
Don't Use LLMs to Make Relevance Judgments
AI总结：
本文基于ACM SIGIR 2024研讨会主题演讲，指出为TREC风格测试集创建相关性判断复杂且昂贵，并明确警告不要使用LLM进行此类判断。
摘要：
为TREC风格的测试集进行相关性判断既复杂又昂贵。一个典型的TREC赛道通常需要六名承包商组成的团队工作2-4周。这些承包商需要接受培训和监督。必须编写软件以支持正确且高效地记录相关性判断。近期，大型语言模型的出现能够响应自然语言提示生成惊人地类似人类的流畅文本输出，这激发了信息检索研究人员思考这些模型如何被用于相关性判断收集过程。在ACM SIGIR 2024会议上，一个名为“LLM4Eval”的研讨会为此类工作提供了场所，并举办了一项数据挑战活动，参与者重现了TREC深度学习赛道的判断，正如Thomas等人所做（arXiv:2408.08896, arXiv:2309.10621）。我被邀请在该研讨会上发表主题演讲，本文以文章形式呈现该主题演讲。核心信息是：不要使用LLM为TREC风格的评估创建相关性判断。
英文摘要：
Making the relevance judgments for a TREC-style test collection can be complex and expensive. A typical TREC track usually involves a team of six contractors working for 2-4 weeks. Those contractors need to be trained and monitored. Software has to be written to support recording relevance judgments correctly and efficiently. The recent advent of large language models that produce astoundingly human-like flowing text output in response to a natural language prompt has inspired IR researchers to wonder how those models might be used in the relevance judgment collection process. At the ACM SIGIR 2024 conference, a workshop ``LLM4Eval'' provided a venue for this work, and featured a data challenge activity where participants reproduced TREC deep learning track judgments, as was done by Thomas et al (arXiv:2408.08896, arXiv:2309.10621). I was asked to give a keynote at the workshop, and this paper presents that keynote in article form. The bottom-line-up-front message is, don't use LLMs to create relevance judgments for TREC-style evaluations.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
