arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2304.08138cs.IR
收藏

面向鲁棒稠密检索的拼写错误感知瓶颈预训练

Typos-aware Bottlenecked Pre-Training for Robust Dense Retrieval

Shengyao Zhuang, Linjun Shou, Jian Pei, Ming Gong, Houxing Ren, Guido Zuccon, Daxin Jiang

AI总结:

针对稠密检索器处理拼写错误查询能力不足的问题,提出一种新型重训练策略ToRoDer,通过编码器-解码器架构进行掩码恢复预训练,显著提升了检索器对拼写错误查询的鲁棒性,同时保持了在正常查询上的检索效果。

摘要:

当前的稠密检索器(DRs)在有效处理拼写错误的查询方面能力有限,而这些查询在商业搜索引擎的查询流量中占有很大比例。主要问题在于,DRs所使用的基于预训练语言模型的编码器,通常使用干净、精心整理的文本数据进行训练和微调。拼写错误的查询通常不存在于用于训练这些模型的数据中,因此,在推理时观察到的拼写错误查询与用于训练和微调的数据相比属于分布外数据。以往解决这一问题的努力主要集中在微调策略上,但它们在拼写错误查询上的有效性仍然低于采用独立的、最先进拼写检查组件的流水线。为了应对这一挑战,我们提出了ToRoDer(TypOs-aware bottlenecked pre-training for RObust DEnse Retrieval),这是一种针对DRs的新型重训练策略,可在保持其在下游检索任务中有效性的同时,提高其对拼写错误查询的鲁棒性。ToRoDer采用编码器-解码器架构,其中编码器接收带有掩码标记的拼写错误文本作为输入,并输出瓶颈信息给解码器。随后,解码器将瓶颈嵌入以及原始文本的标记嵌入(其中拼写错误的标记被掩码)作为输入。预训练任务是恢复编码器和解码器的掩码标记。我们广泛的实验结果和详细的消融研究表明,使用ToRoDer预训练的DRs在拼写错误查询上表现出显著更高的有效性,合理地缩小了与使用独立复杂拼写检查组件流水线之间的差距,同时保留了其在正确拼写查询上的有效性。

英文摘要:

Current dense retrievers (DRs) are limited in their ability to effectively process misspelled queries, which constitute a significant portion of query traffic in commercial search engines. The main issue is that the pre-trained language model-based encoders used by DRs are typically trained and fine-tuned using clean, well-curated text data. Misspelled queries are typically not found in the data used for training these models, and thus misspelled queries observed at inference time are out-of-distribution compared to the data used for training and fine-tuning. Previous efforts to address this issue have focused on \textit{fine-tuning} strategies, but their effectiveness on misspelled queries remains lower than that of pipelines that employ separate state-of-the-art spell-checking components. To address this challenge, we propose ToRoDer (TypOs-aware bottlenecked pre-training for RObust DEnse Retrieval), a novel re-training strategy for DRs that increases their robustness to misspelled queries while preserving their effectiveness in downstream retrieval tasks. ToRoDer utilizes an encoder-decoder architecture where the encoder takes misspelled text with masked tokens as input and outputs bottlenecked information to the decoder. The decoder then takes as input the bottlenecked embeddings, along with token embeddings of the original text with the misspelled tokens masked out. The pre-training task is to recover the masked tokens for both the encoder and decoder. Our extensive experimental results and detailed ablation studies show that DRs pre-trained with ToRoDer exhibit significantly higher effectiveness on misspelled queries, sensibly closing the gap with pipelines that use a separate, complex spell-checker component, while retaining their effectiveness on correctly spelled queries.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑