FECT：联络中心对话记录中解释性AI生成主张的事实性评估
FECT: Factuality Evaluation of Interpretive AI-Generated Claims in Contact Center Conversation Transcripts
AI总结：
针对联络中心对话分析中LLM幻觉问题，提出3D标注范式与FECT基准数据集，用于自动评估解释性AI生成主张的事实性。
摘要：
大型语言模型（LLM）已知会产生幻觉，生成未基于输入、参考资料或现实世界知识的自然语言输出。在企业应用中，当AI功能支持业务决策时，此类幻觉可能尤为有害。分析和总结联络中心对话的LLM为事实性评估引入了独特的挑战，因为对于对话中捕捉到的情绪和业务问题的根本原因的分析性解释，通常不存在真实标签。为解决此问题，我们首先在人工标注指南和LLM评判者的提示中引入了一个三维（3D）范式——分解、解耦、分离——以将事实性标签基于语言学依据的评估标准。接着，我们引入了FECT，一个用于联络中心对话记录中解释性AI生成主张的事实性评估的新型基准数据集，该数据集在我们的3D范式下进行标注。最后，我们报告了在3D范式上对齐LLM评判者的研究结果。总体而言，我们的发现为自动评估用于分析联络中心对话的AI系统生成输出的事实性提供了一种新方法。
英文摘要：
Large language models (LLMs) are known to hallucinate, producing natural language outputs that are not grounded in the input, reference materials, or real-world knowledge. In enterprise applications where AI features support business decisions, such hallucinations can be particularly detrimental. LLMs that analyze and summarize contact center conversations introduce a unique set of challenges for factuality evaluation, because ground-truth labels often do not exist for analytical interpretations about sentiments captured in the conversation and root causes of the business problems. To remedy this, we first introduce a \textbf{3D} -- \textbf{Decompose, Decouple, Detach} -- paradigm in the human annotation guideline and the LLM-judges' prompt to ground the factuality labels in linguistically-informed evaluation criteria. We then introduce \textbf{FECT}, a novel benchmark dataset for \textbf{F}actuality \textbf{E}valuation of Interpretive AI-Generated \textbf{C}laims in Contact Center Conversation \textbf{T}ranscripts, labeled under our 3D paradigm. Lastly, we report our findings from aligning LLM-judges on the 3D paradigm. Overall, our findings contribute a new approach for automatically evaluating the factuality of outputs generated by an AI system for analyzing contact center conversations.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
