arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2512.24366cs.IR
收藏

基于文本的可解释推荐模型的事实一致性研究

On the Factual Consistency of Text-based Explainable Recommendation Models

Sorbonne University(索邦大学);CNRS(法国国家科学研究中心);ISIR(信息智能研究所);AgroParisTech(巴黎国立高等农学院);UMR MIA Paris-Saclay(巴黎萨克雷混合研究单位MIA)

Ben Kabongo, Vincent Guigue

AI总结:

本文研究基于文本的可解释推荐模型的事实一致性,提出评估框架和指标,发现尽管模型在语义相似度上表现良好,但事实一致性指标表现不佳,凸显了事实意识评估的重要性。

摘要:

基于文本的可解释推荐旨在生成自然语言解释以支持物品推荐,以提高用户信任和系统透明度。尽管最近的进展利用大语言模型生成流畅的输出,但一个关键问题仍未被充分探讨:这些解释是否与可用证据事实一致?我们引入了一个综合框架来评估基于文本的可解释推荐系统的真实性。我们设计了一个基于提示的流程,利用大语言模型从评论中提取原子解释性陈述,从而构建一个地面真相,以隔离和聚焦其事实内容。将此流程应用于亚马逊评论数据集的五个类别,我们创建了增强的基准,用于细粒度评估解释质量。我们进一步提出了语句级对齐度量,结合大语言模型和自然语言推理方法,以评估生成解释的事实一致性和相关性。在六个最先进的可解释推荐模型上的广泛实验中,我们发现了一个关键差距:尽管模型在语义相似度得分(BERTScore F1:0.81-0.90)上表现良好,但所有我们的事实性指标揭示出惊人的低表现(基于大语言模型的语句级精度:4.38%-32.88%)。这些发现突显了在可解释推荐中进行事实意识评估的必要性,并为开发更可信的解释系统提供了基础。

英文摘要:

Text-based explainable recommendation aims to generate natural-language explanations that justify item recommendations, to improve user trust and system transparency. Although recent advances leverage LLMs to produce fluent outputs, a critical question remains underexplored: are these explanations factually consistent with the available evidence? We introduce a comprehensive framework for evaluating the factual consistency of text-based explainable recommenders. We design a prompting-based pipeline that uses LLMs to extract atomic explanatory statements from reviews, thereby constructing a ground truth that isolates and focuses on their factual content. Applying this pipeline to five categories from the Amazon Reviews dataset, we create augmented benchmarks for fine-grained evaluation of explanation quality. We further propose statement-level alignment metrics that combine LLM- and NLI-based approaches to assess both factual consistency and relevance of generated explanations. Across extensive experiments on six state-of-the-art explainable recommendation models, we uncover a critical gap: while models achieve high semantic similarity scores (BERTScore F1: 0.81-0.90), all our factuality metrics reveal alarmingly low performance (LLM-based statement-level precision: 4.38%-32.88%). These findings underscore the need for factuality-aware evaluation in explainable recommendation and provide a foundation for developing more trustworthy explanation systems.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑