分析对话摘要中的LLM行为：揭示情境性幻觉趋势
Analyzing LLM Behavior in Dialogue Summarization: Unveiling Circumstantial Hallucination Trends
AI总结：
该研究基准测试了GPT-4和Alpaca-13B在对话摘要中的忠实度，提出“情境推论”错误分类法，并引入两种基于提示的细粒度错误检测方法，优于现有指标。
摘要：
大型语言模型（LLMs）的最新进展显著提升了摘要系统的能力。然而，它们仍面临幻觉问题。尽管先前的工作已在新闻领域对LLMs进行了广泛评估，但大多数对话摘要的评估集中于基于BART的模型，在对其忠实度的理解上留下了空白。我们的工作对LLMs在对话摘要中的忠实度进行了基准测试，使用人工标注并侧重于识别和分类跨度级别的不一致。具体而言，我们关注两个突出的LLMs：GPT-4和Alpaca-13B。我们的评估揭示了关于什么构成幻觉的微妙之处：LLMs经常生成看似合理的推论，这些推论由对话中的情境证据支持，但缺乏直接证据，这种模式在较旧的模型中较少见。我们提出了一个改进的错误分类法，创造了“情境推论”这一类别来归纳这些LLM行为，并发布了数据集。使用我们的分类法，我们比较了LLMs与较旧的微调模型之间的行为差异。此外，我们系统地评估了自动错误检测方法在LLM摘要上的有效性，发现它们难以检测这些细微的错误。为此，我们引入了两种基于提示的细粒度错误检测方法，其表现优于现有指标，特别是在识别“情境推论”方面。
英文摘要：
Recent advancements in large language models (LLMs) have considerably advanced the capabilities of summarization systems. However, they continue to face concerns about hallucinations. While prior work has evaluated LLMs extensively in news domains, most evaluation of dialogue summarization has focused on BART-based models, leaving a gap in our understanding of their faithfulness. Our work benchmarks the faithfulness of LLMs for dialogue summarization, using human annotations and focusing on identifying and categorizing span-level inconsistencies. Specifically, we focus on two prominent LLMs: GPT-4 and Alpaca-13B. Our evaluation reveals subtleties as to what constitutes a hallucination: LLMs often generate plausible inferences, supported by circumstantial evidence in the conversation, that lack direct evidence, a pattern that is less prevalent in older models. We propose a refined taxonomy of errors, coining the category of "Circumstantial Inference" to bucket these LLM behaviors and release the dataset. Using our taxonomy, we compare the behavioral differences between LLMs and older fine-tuned models. Additionally, we systematically assess the efficacy of automatic error detection methods on LLM summaries and find that they struggle to detect these nuanced errors. To address this, we introduce two prompt-based approaches for fine-grained error detection that outperform existing metrics, particularly for identifying "Circumstantial Inference."
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
