arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2403.16416cs.AI
收藏

你的模拟器有多可靠?对当前基于LLM的对话式推荐用户模拟器局限性的分析

How Reliable is Your Simulator? Analysis on the Limitations of Current LLM-based User Simulators for Conversational Recommendation

School of Computer Science and Technology, Beijing Jiaotong University(北京交通大学计算机科学与技术学院);Key Laboratory of Big Data & Artificial Intelligence in Transportation(Beijing Jiaotong University), Ministry of Education(教育部交通大数据与人工智能重点实验室(北京交通大学))

Lixi Zhu, Xiaowen Huang, Jitao Sang

AI总结:

本文分析了基于LLM的对话式推荐用户模拟器的局限性,发现数据泄露、对话历史质量影响及单模板控制困难等问题,并提出SimpleUserSim策略以提升推荐效果。

摘要:

对话式推荐系统(CRS)通过自然语言与用户交互,以理解其偏好并实时提供个性化推荐。CRS已展现出巨大潜力,促使研究人员将开发更真实、更可靠的用户模拟器作为关键焦点。近期,大型语言模型(LLMs)的能力在各领域引起广泛关注。同时,构建基于LLMs的用户模拟器的努力也在进行中。尽管这些工作展示了创新性,但也存在需要关注的某些局限性。在本研究中,我们旨在分析在CRS中使用LLMs构建用户模拟器的局限性,以指导未来研究。为实现这一目标,我们对代表性工作iEvaLM进行了分析验证。通过在对话推荐领域两个广泛使用的数据集上进行多次实验,我们突出了当前基于LLMs的用户模拟器评估方法存在的几个问题:(1)数据泄露,发生在对话历史和用户模拟器的回复中,导致评估结果虚高。(2)CRS推荐的成功更多取决于对话历史的可用性和质量,而非用户模拟器的响应。(3)通过单一提示模板控制用户模拟器的输出具有挑战性。为克服这些局限性,我们提出了SimpleUserSim,采用直接策略将话题引导至目标物品。我们的研究验证了CRS模型利用交互信息的能力,显著改善了推荐结果。

英文摘要:

Conversational Recommender System (CRS) interacts with users through natural language to understand their preferences and provide personalized recommendations in real-time. CRS has demonstrated significant potential, prompting researchers to address the development of more realistic and reliable user simulators as a key focus. Recently, the capabilities of Large Language Models (LLMs) have attracted a lot of attention in various fields. Simultaneously, efforts are underway to construct user simulators based on LLMs. While these works showcase innovation, they also come with certain limitations that require attention. In this work, we aim to analyze the limitations of using LLMs in constructing user simulators for CRS, to guide future research. To achieve this goal, we conduct analytical validation on the notable work, iEvaLM. Through multiple experiments on two widely-used datasets in the field of conversational recommendation, we highlight several issues with the current evaluation methods for user simulators based on LLMs: (1) Data leakage, which occurs in conversational history and the user simulator's replies, results in inflated evaluation results. (2) The success of CRS recommendations depends more on the availability and quality of conversational history than on the responses from user simulators. (3) Controlling the output of the user simulator through a single prompt template proves challenging. To overcome these limitations, we propose SimpleUserSim, employing a straightforward strategy to guide the topic toward the target items. Our study validates the ability of CRS models to utilize the interaction information, significantly improving the recommendation results.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑