扩展医学案例检索任务中的相关性判断：基于多模态大语言模型的方法
Expanding Relevance Judgments for Medical Case-based Retrieval Task with Multimodal LLMs
AI总结：
本研究利用多模态大语言模型Gemini 1.5 Pro扩展医学案例检索的相关性判断，通过结构化提示策略将人工判断数据集扩展37倍，达到0.6的Kappa一致性，验证了MLLM在医学IR评估中的潜力。
摘要：
评估信息检索（IR）系统依赖于高质量的人工相关性判断（qrels），而获取这些判断成本高昂且耗时。虽然池化技术减少了标注工作量，但只能得到部分标注的数据集。大语言模型（LLMs）为减少对人工判断的依赖提供了一种有前景的替代方案，尤其是在医学案例检索等复杂领域，这些领域的相关性评估需要同时分析文本和视觉信息。在本工作中，我们探索使用多模态大语言模型（MLLM）来扩展相关性判断，创建一个自动判断的新数据集。具体而言，我们在ImageCLEFmed 2013案例检索任务上使用Gemini 1.5 Pro，通过一种迭代细化、结构化的提示策略模拟人工评估，该策略整合了二元评分、基于指令的评估和少样本学习。我们系统地实验了多种提示配置，以最大化与人工判断的一致性。为评估MLLM与人工判断之间的一致性，我们使用Cohen's Kappa系数，取得了0.6的实质性一致性分数，与多模态检索任务中通常观察到的标注者间一致性相当。从最初涵盖35个主题的15,028条人工判断（其中4.72%为相关）出发，我们基于MLLM的方法将数据集扩展了超过37倍，达到558,653条判断，相关标注增加到5,950条。平均而言，每个医学案例查询获得了15,398条新标注，其中约99%为非相关，反映了该领域典型的高稀疏性。我们的结果证明了MLLM在扩展相关性判断收集方面的潜力，为支持医学和多模态IR任务中的检索评估提供了一个有前景的方向。
英文摘要：
Evaluating Information Retrieval (IR) systems relies on high-quality manual relevance judgments (qrels), which are costly and time-consuming to obtain. While pooling reduces the annotation effort, it results in only partially labeled datasets. Large Language Models (LLMs) offer a promising alternative to reducing reliance on manual judgments, particularly in complex domains like medical case-based retrieval, where relevance assessment requires analyzing both textual and visual information. In this work, we explore using a Multimodal Large Language Model (MLLM) to expand relevance judgments, creating a new dataset of automated judgments. Specifically, we employ Gemini 1.5 Pro on the ImageCLEFmed 2013 case-based retrieval task, simulating human assessment through an iteratively refined, structured prompting strategy that integrates binary scoring, instruction-based evaluation, and few-shot learning. We systematically experimented with various prompt configurations to maximize agreement with human judgments. To evaluate agreement between the MLLM and human judgments, we use Cohen's Kappa, achieving a substantial agreement score of 0.6, comparable to inter-annotator agreement typically observed in multimodal retrieval tasks. Starting from the original 15,028 manual judgments (4.72% relevant) across 35 topics, our MLLM-based approach expanded the dataset by over 37x to 558,653 judgments, increasing relevant annotations to 5,950. On average, each medical case query received 15,398 new annotations, with approximately 99% being non-relevant, reflecting the high sparsity typical in this domain. Our results demonstrate the potential of MLLMs to scale relevance judgment collection, offering a promising direction for supporting retrieval evaluation in medical and multimodal IR tasks.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
