探究个性化对话式信息检索中LLM的可变性
Investigating LLM Variability in Personalized Conversational Information Retrieval
AI总结：
该研究复现并拓展了此前的个性化对话式信息检索相关工作，通过多模型、多数据集实验验证了LLM输出可变性的影响，指出人工选择PTKB的优势及多轮评估的必要性。
摘要：
近年来，在大语言模型（Large Language Models, LLM）发展的推动下，个性化对话式信息检索（Conversational Information Retrieval, CIR）取得了快速进展。个性化CIR旨在通过利用用户偏好、知识或约束等用户特定信息来优化文档检索，从而根据个体需求定制响应。该任务的一项核心资源是TREC iKAT 2023数据集，其设计目的是评估CIR流水线中的个性化效果。基于这一资源，Mo等人探索了多种将个人文本知识库（Personal Textual Knowledge Bases, PTKB）融入基于LLM的查询重构的策略。他们的研究结果表明，来自PTKB的个性化可能会产生负面影响，且人工标注往往存在噪声。然而，这些结论是基于使用GPT-3.5 Turbo模型的单次运行实验得出的，这引发了人们对输出可变性和可重复性的担忧。在这项可复现性研究中，我们严格复现并拓展了他们的工作，重点关注LLM的输出可变性和模型泛化能力。我们将原始方法应用于新的TREC iKAT 2024数据集，并评估了多种不同的模型，包括Llama（1B-70B）、Qwen-7B、GPT-4o-mini。我们的结果显示，人工选择的PTKB始终能提升检索性能，而基于LLM的选择方法并不能可靠地优于人工选择。我们进一步比较了不同数据集之间的方差，观察到iKAT上的可变性高于CAsT，凸显了评估个性化CIR的挑战。值得注意的是，面向召回率的指标表现出比面向精确率的指标更低的方差，这对第一阶段检索器而言是一项关键洞见。最后，我们强调在评估基于LLM的CIR系统时，需要进行多次运行评估并报告方差。通过在模型、数据集和指标层面拓展评估范围，我们的研究为个性化CIR更稳健、更具泛化性的实践提供了支撑。
英文摘要：
Personalized Conversational Information Retrieval (CIR) has seen rapid progress in recent years, driven by the development of Large Language Models (LLMs). Personalized CIR aims to enhance document retrieval by leveraging user-specific information, such as preferences, knowledge, or constraints, to tailor responses to individual needs. A key resource for this task is the TREC iKAT 2023 dataset, designed to evaluate personalization in CIR pipelines. Building on this resource, Mo et al. explored several strategies for incorporating Personal Textual Knowledge Bases (PTKB) into LLM-based query reformulation. Their findings suggested that personalization from PTKBs could be detrimental and that human annotations were often noisy. However, these conclusions were based on single-run experiments using the GPT-3.5 Turbo model, raising concerns about output variability and repeatability. In this reproducibility study, we rigorously reproduce and extend their work, focusing on LLM output variability and model generalization. We apply the original methods to the new TREC iKAT 2024 dataset and evaluate a diverse range of models, including Llama (1B-70B), Qwen-7B, GPT-4o-mini. Our results show that human-selected PTKBs consistently enhance retrieval performance, while LLM-based selection methods do not reliably outperform manual choices. We further compare variance across datasets and observe higher variability on iKAT than on CAsT, highlighting the challenges of evaluating personalized CIR. Notably, recall-oriented metrics exhibit lower variance than precision-oriented ones, a critical insight for first-stage retrievers. Finally, we underscore the need for multi-run evaluations and variance reporting when assessing LLM-based CIR systems. By broadening evaluation across models, datasets, and metrics, our study contributes to more robust and generalizable practices for personalized CIR.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
