arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2209.06583cs.IRcs.AIcs.CL
收藏

信息检索的前期训练:超链接是否被充分利用?

Pre-training for Information Retrieval: Are Hyperlinks Fully Explored?

Huawei Poisson Lab(华为泊松实验室);University of Montreal(蒙特利尔大学);Renmin University of China(中国人民大学)

Jiawen Wu, Xinyu Zhang, Yutao Zhu, Zheng Liu, Zikai Guo, Zhaoye Fei, Ruofei Lai, Yongkang Wu, Zhao Cao, Zhicheng Dou

摘要:

近年来,在应用预训练的语言模型方面取得了很大进展,例如,BERT,到信息检索(IR)任务。网页中常用的超链接已被用来设计培训前的目标。例如,超链接的锚文本已被用于模拟查询,从而构建了大量的查询文档对进行预训练。然而,作为两个网页之间的桥梁,超链接的潜力还没有得到充分的开发。在这项工作中,我们专注于建模的超链接连接的两个文档之间的关系,并设计一个新的预训练目标的特设检索。具体来说,我们将文档之间的关系分为四组:无链接、单向链接、对称链接和最相关的对称链接。该模型通过对相邻两组文档样本的比较,逐步提高其捕获匹配信号的能力。我们提出了一个渐进的超链接预测({PHP})框架,探索超链接的利用率在预训练。在两个大规模的ad-hoc检索数据集和六个问答数据集上的实验结果表明,该方法优于现有的预训练方法。

英文摘要:

Recent years have witnessed great progress on applying pre-trained language models, e.g., BERT, to information retrieval (IR) tasks. Hyperlinks, which are commonly used in Web pages, have been leveraged for designing pre-training objectives. For example, anchor texts of the hyperlinks have been used for simulating queries, thus constructing tremendous query-document pairs for pre-training. However, as a bridge across two web pages, the potential of hyperlinks has not been fully explored. In this work, we focus on modeling the relationship between two documents that are connected by hyperlinks and designing a new pre-training objective for ad-hoc retrieval. Specifically, we categorize the relationships between documents into four groups: no link, unidirectional link, symmetric link, and the most relevant symmetric link. By comparing two documents sampled from adjacent groups, the model can gradually improve its capability of capturing matching signals. We propose a progressive hyperlink predication ({PHP}) framework to explore the utilization of hyperlinks in pre-training. Experimental results on two large-scale ad-hoc retrieval datasets and six question-answering datasets demonstrate its superiority over existing pre-training methods.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑