QUERT:面向旅游领域搜索中查询理解的语言模型持续预训练
QUERT: Continual Pre-training of Language Model for Query Understanding in Travel Domain Search
AI总结:
该研究提出面向旅游领域搜索查询理解的持续预训练模型QUERT,通过四项定制预训练任务联合训练,在下游任务及飞猪APP线上A/B测试中均实现性能提升。
摘要:
鉴于预训练语言模型(PLMs)的成功,通用预训练语言模型的持续预训练已成为领域自适应的主流范式。本文提出了QUERT——一种面向旅游领域搜索中查询理解的持续预训练语言模型。QUERT结合旅游领域搜索的查询特点,在四项定制化预训练任务上联合训练:地理感知掩码预测、Geohash编码预测、用户点击行为学习,以及短语与词元顺序预测。下游任务的性能提升与消融实验验证了所提预训练任务的有效性。具体而言,在有监督和无监督设置下,下游任务的平均性能分别提升了2.02%和30.93%。为检验QUERT对线上业务的提升效果,我们部署了QUERT并在飞猪APP上开展A/B测试。反馈结果显示,将QUERT用作编码器时,独立点击率和页面点击率分别提升了0.89%和1.03%。我们的代码与下游任务数据将公开发布以供后续研究使用。
英文摘要:
In light of the success of the pre-trained language models (PLMs), continual pre-training of generic PLMs has been the paradigm of domain adaption. In this paper, we propose QUERT, A Continual Pre-trained Language Model for QUERy Understanding in Travel Domain Search. QUERT is jointly trained on four tailored pre-training tasks to the characteristics of query in travel domain search: Geography-aware Mask Prediction, Geohash Code Prediction, User Click Behavior Learning, and Phrase and Token Order Prediction. Performance improvement of downstream tasks and ablation experiment demonstrate the effectiveness of our proposed pre-training tasks. To be specific, the average performance of downstream tasks increases by 2.02% and 30.93% in supervised and unsupervised settings, respectively. To check on the improvement of QUERT to online business, we deploy QUERT and perform A/B testing on Fliggy APP. The feedback results show that QUERT increases the Unique Click-Through Rate and Page Click-Through Rate by 0.89% and 1.03% when applying QUERT as the encoder. Our code and downstream task data will be released for future research.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。