基于LLM的相关性判断中提示词敏感性的人机对比分析
A Human-AI Comparative Analysis of Prompt Sensitivity in LLM-Based Relevance Judgment
AI总结：
该研究针对基于LLM的相关性判断的稳健性，系统调查提示词敏感性影响，通过收集并对比人类与LLM生成的提示词在三类任务中的表现，为相关研究提供数据支持。
摘要：
大型语言模型（LLMs）正越来越多地用于自动化信息检索（IR）任务的相关性判断，其与人类标签的一致性往往接近人际一致性。为评估基于LLM的相关性判断的稳健性和可靠性，我们系统研究了提示词敏感性对该任务的影响。我们从15名人类专家和15个LLM处收集了三种任务（二元、分级和成对）的相关性评估提示词，共得到90个提示词。在过滤掉3名人类和3个LLM的不可用提示词后，我们使用剩余的72个提示词，以三种不同的LLM作为判断者，对两个TREC深度学习数据集（2020和2021）的文档/查询对进行标注。我们使用Cohen's κ和成对一致性度量将LLM生成的标签与TREC官方人类标签进行比较。除了研究提示词变化对与人类标签一致性的影响外，我们还比较了人类和LLM生成的提示词，并分析了不同LLM作为判断者之间的差异。我们还将人类和LLM生成的提示词与Bing和TREC 2024检索增强生成（RAG）赛道用于相关性评估的标准UMBRELA提示词进行了比较。为支持未来基于LLM的评估研究，我们在https://github.com/Narabzad/prompt-sensitivity-relevance-judgements/上发布了所有数据和提示词。
英文摘要：
Large Language Models (LLMs) are increasingly used to automate relevance judgments for information retrieval (IR) tasks, often demonstrating agreement with human labels that approaches inter-human agreement. To assess the robustness and reliability of LLM-based relevance judgments, we systematically investigate impact of prompt sensitivity on the task. We collected prompts for relevance assessment from 15 human experts and 15 LLMs across three tasks~ -- ~binary, graded, and pairwise~ -- ~yielding 90 prompts in total. After filtering out unusable prompts from three humans and three LLMs, we employed the remaining 72 prompts with three different LLMs as judges to label document/query pairs from two TREC Deep Learning Datasets (2020 and 2021). We compare LLM-generated labels with TREC official human labels using Cohen's $κ$ and pairwise agreement measures. In addition to investigating the impact of prompt variations on agreement with human labels, we compare human- and LLM-generated prompts and analyze differences among different LLMs as judges. We also compare human- and LLM-generated prompts with the standard UMBRELA prompt used for relevance assessment by Bing and TREC 2024 Retrieval Augmented Generation (RAG) Track. To support future research in LLM-based evaluation, we release all data and prompts at https://github.com/Narabzad/prompt-sensitivity-relevance-judgements/.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
