我们能否使用大型语言模型填补相关性判断漏洞？
Can We Use Large Language Models to Fill Relevance Judgment Holes?
AI总结：
本文研究用 LLM 基于既有判断填补测试集相关性漏洞，发现仅补洞会造成排名偏差，全文档池标注更一致。
摘要：
不完整的相关性判断限制了测试集的可复用性。当新系统与曾用于构建已判断文档池的既往系统进行比较时，由于测试集中的“漏洞”（即新系统返回的成批未评估文档），新系统往往处于不利地位。本文中，我们采取初步措施来扩展现有的测试集：利用大型语言模型（LLM）填补这些漏洞，并借助现有的人工判断为该方法提供依据和基础。我们在使用 TREC iKAT 的对话式搜索场景下探讨这一问题；在该场景中，信息需求高度动态，响应（以及检索到的结果）变化更大，从而留下更大的漏洞。尽管既往工作表明，LLM 给出的自动判断会产生高度相关的排名，但我们发现，当同时使用人工判断和自动判断时，相关性会显著降低，无论使用何种 LLM、采用单样本/双样本/少样本方式，还是进行微调，结果都是如此。我们进一步发现，根据所采用的 LLM，新的运行结果会受到明显偏好（或惩罚），而且这种效应会随着漏洞规模成比例放大。相反，应当在整个文档池上生成 LLM 标注，以获得与人工生成标签更一致的排名。未来需要开展提示工程和 LLM 微调工作，使其反映并表征人工标注，从而为模型提供依据并实现对齐，使其更符合用途。
英文摘要：
Incomplete relevance judgments limit the re-usability of test collections. When new systems are compared against previous systems used to build the pool of judged documents, they often do so at a disadvantage due to the ``holes'' in test collection (i.e., pockets of un-assessed documents returned by the new system). In this paper, we take initial steps towards extending existing test collections by employing Large Language Models (LLM) to fill the holes by leveraging and grounding the method using existing human judgments. We explore this problem in the context of Conversational Search using TREC iKAT, where information needs are highly dynamic and the responses (and, the results retrieved) are much more varied (leaving bigger holes). While previous work has shown that automatic judgments from LLMs result in highly correlated rankings, we find substantially lower correlates when human plus automatic judgments are used (regardless of LLM, one/two/few shot, or fine-tuned). We further find that, depending on the LLM employed, new runs will be highly favored (or penalized), and this effect is magnified proportionally to the size of the holes. Instead, one should generate the LLM annotations on the whole document pool to achieve more consistent rankings with human-generated labels. Future work is required to prompt engineering and fine-tuning LLMs to reflect and represent the human annotations, in order to ground and align the models, such that they are more fit for purpose.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
