关于大语言模型用于相关性判断的观点
Perspectives on Large Language Models for Relevance Judgment
AI总结：
本文探讨大语言模型在相关性判断中的应用，提出人机协作频谱，并通过初步实验及文献分析，给出支持、反对及折中的观点。
摘要：
当被问及时，像ChatGPT这样的大语言模型（LLMs）声称它们可以协助进行相关性判断，但目前尚不清楚自动判断能否可靠地用于检索系统的评估。在这篇观点论文中，我们讨论了LLMs支持相关性判断的可能方式，以及由此产生的担忧和问题。我们设计了一个人机协作频谱，该频谱基于人类对机器的依赖程度，对不同的相关性判断策略进行分类。对于“全自动判断”这一极端点，我们进一步进行了一项初步实验，探讨基于LLM的相关性判断是否与经过训练的人类评估者的判断相关。最后，我们通过提供支持和反对使用LLMs进行自动相关性判断的对立观点，以及一个基于我们对文献的分析、初步实验证据以及我们作为信息检索研究者的经验而形成的折中观点，来总结本文。
英文摘要：
When asked, large language models (LLMs) like ChatGPT claim that they can assist with relevance judgments but it is not clear whether automated judgments can reliably be used in evaluations of retrieval systems. In this perspectives paper, we discuss possible ways for LLMs to support relevance judgments along with concerns and issues that arise. We devise a human--machine collaboration spectrum that allows to categorize different relevance judgment strategies, based on how much humans rely on machines. For the extreme point of "fully automated judgments", we further include a pilot experiment on whether LLM-based relevance judgments correlate with judgments from trained human assessors. We conclude the paper by providing opposing perspectives for and against the use of~LLMs for automatic relevance judgments, and a compromise perspective, informed by our analyses of the literature, our preliminary experimental evidence, and our experience as IR researchers.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
