arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2108.09346cs.IR
收藏

Ad-hoc检索的预训练:超链接也是你需要的

Pre-training for Ad-hoc Retrieval: Hyperlink is Also You Need

Zhengyi Ma, Zhicheng Dou, Wei Xu, Xinyu Zhang, Hao Jiang, Zhao Cao, Ji-Rong Wen

摘要:

为预训练的语言模型设计更接近下游任务的预训练目标可以在微调阶段获得更好的性能,特别是在ad-hoc检索领域。现有的预训练方法为IR量身定制,试图将弱监督信号,如查询可能性为基础的采样,从原始文本语料库构建伪查询文档对。然而,这些信号严重依赖于采样方法。例如,查询似然模型可能导致在构造的预训练数据中的很多噪声。\blfootnote{$\dagger$ This work was done during an internship at Huawei.}在本文中,我们提出利用大规模的超链接和锚文本来预训练用于ad-hoc检索的语言模型。由于锚文本是由网站管理员创建的,通常可以总结目标文档,因此它可以帮助构建比特定算法更准确和可靠的预训练样本。考虑到下游ad-hoc检索的不同观点,我们设计了四个基于超链接的预训练任务。然后,我们对Transformer模型进行预训练,以预测成对偏好,并结合Masked语言模型目标。在两个大规模ad-hoc检索数据集上的实验结果表明,与现有方法相比,该模型有显著的改进.

英文摘要:

Designing pre-training objectives that more closely resemble the downstream tasks for pre-trained language models can lead to better performance at the fine-tuning stage, especially in the ad-hoc retrieval area. Existing pre-training approaches tailored for IR tried to incorporate weak supervised signals, such as query-likelihood based sampling, to construct pseudo query-document pairs from the raw textual corpus. However, these signals rely heavily on the sampling method. For example, the query likelihood model may lead to much noise in the constructed pre-training data. \blfootnote{$\dagger$ This work was done during an internship at Huawei.} In this paper, we propose to leverage the large-scale hyperlinks and anchor texts to pre-train the language model for ad-hoc retrieval. Since the anchor texts are created by webmasters and can usually summarize the target document, it can help to build more accurate and reliable pre-training samples than a specific algorithm. Considering different views of the downstream ad-hoc retrieval, we devise four pre-training tasks based on the hyperlinks. We then pre-train the Transformer model to predict the pair-wise preference, jointly with the Masked Language Model objective. Experimental results on two large-scale ad-hoc retrieval datasets show the significant improvement of our model compared with the existing methods.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑