大型语言模型作为推荐解释的评估器
Large Language Models as Evaluators for Recommendation Explanations
AI总结：
本文研究大型语言模型能否作为推荐解释的评估器，通过3级元评估策略和真实用户反馈验证了GPT4等LLM在适当提示下可提供准确、可复现且低成本的评估，并提出结合人类标签与多模型集成以提升评估稳定性。
摘要：
推荐系统的可解释性在学术界和工业界引起了广泛关注。尽管在可解释推荐方面已经做出了许多努力，但评估解释质量仍然是一个具有挑战性且尚未解决的问题。近年来，利用大型语言模型（LLMs）作为评估器在自然语言处理任务（例如情感分类、信息抽取）中展现出了有前景的途径，因为它们在指令遵循和常识推理方面表现出强大的能力。然而，评估推荐解释文本与这些自然语言生成（NLG）任务不同，因为其标准与人类感知相关，并且通常是主观的。在本文中，我们研究LLMs是否可以作为推荐解释的评估器。为了回答这个问题，我们利用先前工作中用户对解释的真实反馈，并额外收集第三方标注和LLM评估。我们设计并应用了一个3级元评估策略，以衡量评估器标签与用户提供的真实标签之间的相关性。我们的实验表明，诸如GPT4之类的LLMs在适当的提示和设置下可以提供可比的评估。我们还进一步提供了关于将人类标签与LLM评估过程相结合以及利用多个异构LLM评估器的集成来提高评估准确性和稳定性的见解。我们的研究验证了利用LLMs作为评估器可以成为评估推荐解释文本的一种准确、可复现且成本效益高的解决方案。我们的代码可在https://github.com/Xiaoyu-SZ/LLMasEvaluator获取。
英文摘要：
The explainability of recommender systems has attracted significant attention in academia and industry. Many efforts have been made for explainable recommendations, yet evaluating the quality of the explanations remains a challenging and unresolved issue. In recent years, leveraging LLMs as evaluators presents a promising avenue in Natural Language Processing tasks (e.g., sentiment classification, information extraction), as they perform strong capabilities in instruction following and common-sense reasoning. However, evaluating recommendation explanatory texts is different from these NLG tasks, as its criteria are related to human perceptions and are usually subjective. In this paper, we investigate whether LLMs can serve as evaluators of recommendation explanations. To answer the question, we utilize real user feedback on explanations given from previous work and additionally collect third-party annotations and LLM evaluations. We design and apply a 3-level meta evaluation strategy to measure the correlation between evaluator labels and the ground truth provided by users. Our experiments reveal that LLMs, such as GPT4, can provide comparable evaluations with appropriate prompts and settings. We also provide further insights into combining human labels with the LLM evaluation process and utilizing ensembles of multiple heterogeneous LLM evaluators to enhance the accuracy and stability of evaluations. Our study verifies that utilizing LLMs as evaluators can be an accurate, reproducible and cost-effective solution for evaluating recommendation explanation texts. Our code is available at https://github.com/Xiaoyu-SZ/LLMasEvaluator.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
