arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2109.01274cs.IR
收藏

UserBERT:对比用户模型预训练

UserBERT: Contrastive User Model Pre-training

Chuhan Wu, Fangzhao Wu, Yang Yu, Tao Qi, Yongfeng Huang, Xing Xie

摘要:

用户建模对于个性化Web应用程序至关重要。现有的用户建模方法通常使用特定于任务的标记数据从用户行为中训练用户模型。然而,目标任务中的标记数据可能不足以训练准确的用户模型。幸运的是,通常存在丰富的未标记的用户行为数据,这些数据编码了丰富的用户特征和兴趣信息。因此,在未标记的用户行为数据上预训练用户模型有可能改善许多下游任务的用户建模。在本文中,我们提出了一种对比用户模型预训练方法命名为UserBERT。UserBERT中包含两个自我监督任务,用于在未标记的用户行为数据上进行用户模型预训练,以增强用户建模能力。第一个是屏蔽行为预测,旨在对用户行为之间的相关性进行建模。第二种是行为序列匹配,其目的是捕捉在不同时期具有一致性的用户内在兴趣。此外,我们提出了一个中等难度的负样本框架来选择信息丰富的负样本,以进行更好的对比预训练。我们维护一个同步更新的候选行为池和一个异步更新的候选行为序列池,以有效地选择本地最难的负面行为和行为序列。在两个不同任务的真实数据集上的大量实验表明,UserBERT可以有效地改善各种用户模型。

英文摘要:

User modeling is critical for personalized web applications. Existing user modeling methods usually train user models from user behaviors with task-specific labeled data. However, labeled data in a target task may be insufficient for training accurate user models. Fortunately, there are usually rich unlabeled user behavior data which encode rich information of user characteristics and interests. Thus, pre-training user models on unlabeled user behavior data has the potential to improve user modeling for many downstream tasks. In this paper, we propose a contrastive user model pre-training method named UserBERT. Two self-supervision tasks are incorporated in UserBERT for user model pre-training on unlabeled user behavior data to empower user modeling. The first one is masked behavior prediction, which aims to model the relatedness between user behaviors. The second one is behavior sequence matching, which aims to capture the inherent user interests that are consistent in different periods. In addition, we propose a medium-hard negative sampling framework to select informative negative samples for better contrastive pre-training. We maintain a synchronously updated candidate behavior pool and an asynchronously updated candidate behavior sequence pool to select the locally hardest negative behaviors and behavior sequences in an efficient way. Extensive experiments on two real-world datasets in different tasks show that UserBERT can effectively improve various user models.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑