PTUM:通过自我监督从未标记的用户行为预训练用户模型
PTUM: Pre-training User Model from Unlabeled User Behaviors via Self-supervision
摘要:
用户建模对于许多个性化的Web服务至关重要。现有的方法大多基于用户的行为和目标任务的标记数据对用户进行建模。然而,这些方法不能利用未标记的用户行为数据中的有用信息,并且当标记数据稀缺时,它们的性能可能不是最佳的。基于在大规模未标记语料库上预训练的语言模型,本文提出从大规模未标记用户行为数据中预训练用户模型。我们提出了两个自我监督的用户模型预训练任务。第一种是掩蔽行为预测,它可以模拟历史行为之间的相关性。第二个是下一个$K$行为预测,它可以模拟过去和未来行为之间的相关性。预先训练的用户模型在下游任务中进行微调,以学习特定于任务的用户表示。在两个真实数据集上的实验结果验证了我们提出的用户模型预训练方法的有效性。
英文摘要:
User modeling is critical for many personalized web services. Many existing methods model users based on their behaviors and the labeled data of target tasks. However, these methods cannot exploit useful information in unlabeled user behavior data, and their performance may be not optimal when labeled data is scarce. Motivated by pre-trained language models which are pre-trained on large-scale unlabeled corpus to empower many downstream tasks, in this paper we propose to pre-train user models from large-scale unlabeled user behaviors data. We propose two self-supervision tasks for user model pre-training. The first one is masked behavior prediction, which can model the relatedness between historical behaviors. The second one is next $K$ behavior prediction, which can model the relatedness between past and future behaviors. The pre-trained user models are finetuned in downstream tasks to learn task-specific user representations. Experimental results on two real-world datasets validate the effectiveness of our proposed user model pre-training method.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。