arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2209.00218cs.CL
收藏

各向同性表示可以改善密集检索

Isotropic Representation Can Improve Dense Retrieval

Seoul National University(首尔国立大学);Naver Corp.(NAVER公司)

Euna Jung, Jungwon Park, Jaekeol Choi, Sungyoon Kim, Wonjong Rhee

AI总结:

针对BERT表示的各向异性分布问题,提出Normalizing Flow和whitening等无监督后处理方法,有效提升密集检索模型的性能与鲁棒性。

摘要:

语言表示建模的最新进展广泛影响了密集检索(dense retrieval)模型的设计。具体而言,许多高性能的密集检索模型使用BERT评估查询和文档的表示,随后应用基于余弦相似度的评分来确定相关性。然而,已知BERT表示遵循窄锥形的各向异性分布,这种各向异性分布对于基于余弦相似度的评分是不利的。在本研究中,我们首先表明基于BERT的密集检索模型也遵循各向异性分布。为了应对这一问题,我们引入了Normalizing Flow和whitening的无监督后处理方法,并开发了token-wise方法,以补充sequence-wise方法,将这些后处理方法应用于密集检索模型的表示。我们表明,所提出的方法能够有效地增强表示使其具有各向同性,随后我们使用ColBERT和RepBERT进行实验,证明文档重排序的性能(NDCG at 10)对于ColBERT可提高5.17%至8.09%,对于RepBERT可提高6.88%至22.81%。为了考察各向同性表示在改善密集检索模型鲁棒性方面的潜力,我们研究了测试数据集与训练数据集不同的分布外任务。结果表明,各向同性表示能够实现普遍的性能提升。例如,当训练数据集为MS-MARCO且测试数据集为Robust04时,各向同性后处理可将基线性能提高高达24.98%。此外,我们表明,使用分布外数据集训练的各向同性模型甚至可以超越使用分布内数据集训练的基线模型。

英文摘要:

The recent advancement in language representation modeling has broadly affected the design of dense retrieval models. In particular, many of the high-performing dense retrieval models evaluate representations of query and document using BERT, and subsequently apply a cosine-similarity based scoring to determine the relevance. BERT representations, however, are known to follow an anisotropic distribution of a narrow cone shape and such an anisotropic distribution can be undesirable for the cosine-similarity based scoring. In this work, we first show that BERT-based DR also follows an anisotropic distribution. To cope with the problem, we introduce unsupervised post-processing methods of Normalizing Flow and whitening, and develop token-wise method in addition to the sequence-wise method for applying the post-processing methods to the representations of dense retrieval models. We show that the proposed methods can effectively enhance the representations to be isotropic, then we perform experiments with ColBERT and RepBERT to show that the performance (NDCG at 10) of document re-ranking can be improved by 5.17\%$\sim$8.09\% for ColBERT and 6.88\%$\sim$22.81\% for RepBERT. To examine the potential of isotropic representation for improving the robustness of DR models, we investigate out-of-distribution tasks where the test dataset differs from the training dataset. The results show that isotropic representation can achieve a generally improved performance. For instance, when training dataset is MS-MARCO and test dataset is Robust04, isotropy post-processing can improve the baseline performance by up to 24.98\%. Furthermore, we show that an isotropic model trained with an out-of-distribution dataset can even outperform a baseline model trained with the in-distribution dataset.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑