重新思考基于检索增强的个性化对话的评估：认知与语言学视角
Rethinking Evaluation in Retrieval-Augmented Personalized Dialogue: A Cognitive and Linguistic Perspective
AI总结：
本文从认知与语言学角度重新审视基于检索增强的个性化对话评估，指出传统表面相似度指标无法反映对话质量的深层特征，并通过LAPDOG框架分析，强调需建立更符合人类沟通原理的评估方法。
摘要：
在认知科学和语言理论中，对话被视为由连贯性、一致性及共享理解维持的共同活动，而非独立语句的序列。然而，许多开放领域和个性化对话系统主要依赖表面相似度指标（如BLEU、ROUGE、F1）进行评估，这无法捕捉对话质量的深层特征。本文重新审视一个著名的基于检索增强的个性化对话框架LAPDOG，作为评估方法的案例研究。通过人类和LLM基于的评判者，我们识别出当前评估实践的局限性，包括对话历史的损坏、检索故事与角色之间的矛盾以及不连贯的响应生成。我们的结果表明，人类和LLM的判断高度一致，但与词汇相似度指标存在分歧，强调了需要基于认知的评估方法。广泛而言，本文为更可靠评估框架铺平道路，以更符合自然人类沟通原则的方式评估基于检索增强的对话系统。
英文摘要：
In cognitive science and linguistic theory, dialogue is not seen as a chain of independent utterances but rather as a joint activity sustained by coherence, consistency, and shared understanding. However, many systems for open-domain and personalized dialogue use surface-level similarity metrics (e.g., BLEU, ROUGE, F1) as one of their main reporting measures, which fail to capture these deeper aspects of conversational quality. We re-examine a notable retrieval-augmented framework for personalized dialogue, LAPDOG, as a case study for evaluation methodology. Using both human and LLM-based judges, we identify limitations in current evaluation practices, including corrupted dialogue histories, contradictions between retrieved stories and persona, and incoherent response generation. Our results show that human and LLM judgments align closely but diverge from lexical similarity metrics, underscoring the need for cognitively grounded evaluation methods. Broadly, this work charts a path toward more reliable assessment frameworks for retrieval-augmented dialogue systems that better reflect the principles of natural human communication.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
