arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2106.03373cs.IR
收藏

基于预训练的语言模型在百度搜索中的应用

Pre-trained Language Model for Web-scale Retrieval in Baidu Search

Yiding Liu, Guan Huang, Jiaxiang Liu, Weixue Lu, Suqi Cheng, Yukun Li, Daiting Shi, Shuaiqiang Wang, Zhicong Cheng, Dawei Yin

摘要:

检索是网络搜索中的一个关键阶段,它从十亿规模的语料库中识别出一小部分与查询相关的候选者。在检索阶段发现更多的语义相关的候选是非常有希望向最终用户展示更多的高质量的结果。然而,在实际的搜索引擎中,如何建立和部署有效的语义匹配检索模型仍然是一个不容忽视的挑战。在本文中,我们描述了检索系统,我们开发和部署在百度搜索。该系统利用了最新的国家的最先进的中文预训练的语言模型,即增强表示通过知识整合(ERNIE),这有利于系统的表达语义匹配。特别是,我们开发了一个基于ERNIE的检索模型,该模型配备了1)基于表达性转换器的语义编码器,以及2)全面的多阶段训练范式。更重要的是,我们提出了一个实用的系统工作流程部署模型在网络规模的检索。最终,该系统完全部署到生产环境中,并进行了严格的离线和在线实验。实验结果表明,该系统能够进行高质量的候选检索,特别是对于那些不常见的需求尾查询。总的来说,由预训练的语言模型(即,ERNIE)可以在很大程度上提高我们的搜索引擎的可用性和适用性。

英文摘要:

Retrieval is a crucial stage in web search that identifies a small set of query-relevant candidates from a billion-scale corpus. Discovering more semantically-related candidates in the retrieval stage is very promising to expose more high-quality results to the end users. However, it still remains non-trivial challenges of building and deploying effective retrieval models for semantic matching in real search engine. In this paper, we describe the retrieval system that we developed and deployed in Baidu Search. The system exploits the recent state-of-the-art Chinese pretrained language model, namely Enhanced Representation through kNowledge IntEgration (ERNIE), which facilitates the system with expressive semantic matching. In particular, we developed an ERNIE-based retrieval model, which is equipped with 1) expressive Transformer-based semantic encoders, and 2) a comprehensive multi-stage training paradigm. More importantly, we present a practical system workflow for deploying the model in web-scale retrieval. Eventually, the system is fully deployed into production, where rigorous offline and online experiments were conducted. The results show that the system can perform high-quality candidate retrieval, especially for those tail queries with uncommon demands. Overall, the new retrieval system facilitated by pretrained language model (i.e., ERNIE) can largely improve the usability and applicability of our search engine.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑