从摘要视角评估LLMs对混合上下文幻觉的判定
Evaluating LLMs' Assessment of Mixed-Context Hallucination Through the Lens of Summarization
AI总结：
本研究以摘要为任务，评估LLMs检测混合上下文幻觉的能力，发现其内在知识会引入偏差，导致事实性幻觉检测遇瓶颈，根本挑战在于平衡内在知识与外部上下文。
摘要：
随着大语言模型（LLMs）的快速发展，LLM-as-a-judge已成为一种广泛采用的文本质量评估方法，包括幻觉评估。以往研究仅专注于单上下文评估（如话语忠实度或世界事实性），而现实世界的幻觉通常涉及混合上下文，这仍缺乏充分评估。本研究以摘要作为代表性任务，全面评估LLMs检测混合上下文幻觉的能力，特别是区分事实性与非事实性幻觉。通过对不同规模的直接生成模型和基于检索的模型进行广泛实验，主要观察结果为：(1) LLMs的内在知识在幻觉评估中引入了固有偏差；(2) 这些偏差特别影响事实性幻觉的检测，导致显著的性能瓶颈；(3) 根本挑战在于有效的知识利用，即在LLMs的内在知识与外部上下文之间取得平衡，以实现准确的混合上下文幻觉评估。
英文摘要：
With the rapid development of large language models (LLMs), LLM-as-a-judge has emerged as a widely adopted approach for text quality evaluation, including hallucination evaluation. While previous studies have focused exclusively on single-context evaluation (e.g., discourse faithfulness or world factuality), real-world hallucinations typically involve mixed contexts, which remains inadequately evaluated. In this study, we use summarization as a representative task to comprehensively evaluate LLMs' capability in detecting mixed-context hallucinations, specifically distinguishing between factual and non-factual hallucinations. Through extensive experiments across direct generation and retrieval-based models of varying scales, our main observations are: (1) LLMs' intrinsic knowledge introduces inherent biases in hallucination evaluation; (2) These biases particularly impact the detection of factual hallucinations, yielding a significant performance bottleneck; (3) The fundamental challenge lies in effective knowledge utilization, balancing between LLMs' intrinsic knowledge and external context for accurate mixed-context hallucination evaluation.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
