arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2403.09142cs.IRcs.AI
收藏

USimAgent:用于模拟搜索用户的大语言模型

USimAgent: Large Language Models for Simulating Search Users

Renmin University of China(中国人民大学)

Erhan Zhang, Xingzhu Wang, Peiyuan Gong, Yankai Lin, Jiaxin Mao

AI总结:

针对信息检索系统用户评估中搜索行为模拟的难题,提出基于大语言模型的用户搜索行为模拟器USimAgent,可生成完整搜索会话,在查询生成上优于现有方法,为通用用户模拟器开发提供新思路。

摘要:

凭借成本效益与可复现性方面的优势,用户模拟已成为信息检索系统以用户为中心的评估的一种颇具前景的方案。尽管如此,精准模拟用户搜索行为长期以来一直是一项挑战,因为用户在搜索中的行为极为复杂,且由学习、推理、规划等复杂认知过程驱动。近年来,大语言模型(LLMs)在模拟人类水平智能方面展现出显著潜力,已被用于构建面向各类任务的自主智能体。然而,利用LLMs模拟搜索行为的潜力尚未得到充分挖掘。本文提出了一种基于LLM的用户搜索行为模拟器USimAgent。该模拟器能够模拟用户在搜索过程中的查询、点击和停止行为,因此可为特定搜索任务生成完整的搜索会话。在真实用户行为数据集上的实证研究表明,所提模拟器在查询生成方面优于现有方法,在预测用户点击和停止行为方面与传统方法表现相当。这些结果不仅验证了将LLMs用于用户模拟的有效性,也为开发更稳健、更通用的用户模拟器提供了思路。代码与数据可通过https://github.com/Meow-E/USimAgent获取。

英文摘要:

Due to the advantages in the cost-efficiency and reproducibility, user simulation has become a promising solution to the user-centric evaluation of information retrieval systems. Nonetheless, accurately simulating user search behaviors has long been a challenge, because users' actions in search are highly complex and driven by intricate cognitive processes such as learning, reasoning, and planning. Recently, Large Language Models (LLMs) have demonstrated remarked potential in simulating human-level intelligence and have been used in building autonomous agents for various tasks. However, the potential of using LLMs in simulating search behaviors has not yet been fully explored. In this paper, we introduce a LLM-based user search behavior simulator, USimAgent. The proposed simulator can simulate users' querying, clicking, and stopping behaviors during search, and thus, is capable of generating complete search sessions for specific search tasks. Empirical investigation on a real user behavior dataset shows that the proposed simulator outperforms existing methods in query generation and is comparable to traditional methods in predicting user clicks and stopping behaviors. These results not only validate the effectiveness of using LLMs for user simulation but also shed light on the development of a more robust and generic user simulators. The code and data are accessible at https://github.com/Meow-E/USimAgent.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑