语言表示建模的最新进展广泛影响了密集检索(dense retrieval)模型的设计。具体而言,许多高性能的密集检索模型使用BERT评估查询和文档的表示,随后应用基于余弦相似度的评分来确定相关性。然而,已知BERT表示遵循窄锥形的各向异性分布,这种各向异性分布对于基于余弦相似度的评分是不利的。在本研究中,我们首先表明基于BERT的密集检索模型也遵循各向异性分布。为了应对这一问题,我们引入了Normalizing Flow和whitening的无监督后处理方法,并开发了token-wise方法,以补充sequence-wise方法,将这些后处理方法应用于密集检索模型的表示。我们表明,所提出的方法能够有效地增强表示使其具有各向同性,随后我们使用ColBERT和RepBERT进行实验,证明文档重排序的性能(NDCG at 10)对于ColBERT可提高5.17%至8.09%,对于RepBERT可提高6.88%至22.81%。为了考察各向同性表示在改善密集检索模型鲁棒性方面的潜力,我们研究了测试数据集与训练数据集不同的分布外任务。结果表明,各向同性表示能够实现普遍的性能提升。例如,当训练数据集为MS-MARCO且测试数据集为Robust04时,各向同性后处理可将基线性能提高高达24.98%。此外,我们表明,使用分布外数据集训练的各向同性模型甚至可以超越使用分布内数据集训练的基线模型。
英文摘要:
The recent advancement in language representation modeling has broadly affected the design of dense retrieval models. In particular, many of the high-performing dense retrieval models evaluate representations of query and document using BERT, and subsequently apply a cosine-similarity based scoring to determine the relevance. BERT representations, however, are known to follow an anisotropic distribution of a narrow cone shape and such an anisotropic distribution can be undesirable for the cosine-similarity based scoring. In this work, we first show that BERT-based DR also follows an anisotropic distribution. To cope with the problem, we introduce unsupervised post-processing methods of Normalizing Flow and whitening, and develop token-wise method in addition to the sequence-wise method for applying the post-processing methods to the representations of dense retrieval models. We show that the proposed methods can effectively enhance the representations to be isotropic, then we perform experiments with ColBERT and RepBERT to show that the performance (NDCG at 10) of document re-ranking can be improved by 5.17\%$\sim$8.09\% for ColBERT and 6.88\%$\sim$22.81\% for RepBERT. To examine the potential of isotropic representation for improving the robustness of DR models, we investigate out-of-distribution tasks where the test dataset differs from the training dataset. The results show that isotropic representation can achieve a generally improved performance. For instance, when training dataset is MS-MARCO and test dataset is Robust04, isotropy post-processing can improve the baseline performance by up to 24.98\%. Furthermore, we show that an isotropic model trained with an out-of-distribution dataset can even outperform a baseline model trained with the in-distribution dataset.
Reduce Catastrophic Forgetting of Dense Retrieval Training with Teleportation Negatives
Tsinghua University(清华大学);Microsoft Research(微软研究院);Georgia Institute of Technology(佐治亚理工学院);Beijing National Research Center for Information Science and Technology(北京信息科学与技术国家研究中心)
Answering Complex Open-Domain Questions with Multi-Hop Dense Retrieval
University of California, Santa Barbara(加利福尼亚大学圣巴巴拉分校);Facebook AI(脸书人工智能研究院);University of Massachusetts Amherst(马萨诸塞大学阿默斯特分校);University College London(伦敦大学学院)
Llama2Vec: Unsupervised Adaptation of Large Language Models for Dense Retrieval
Beijing University of Posts and Telecommunications(北京邮电大学);Beijing Academy of Artificial Intelligence(北京人工智能研究院);University of Science and Technology of China(中国科学技术大学);The Hong Kong Polytechnic University(香港理工大学)
When retrieval outperforms generation: Dense evidence retrieval for scalable fake news detection
School of Computer Science, University of Galway, Ireland(Galway大学计算机科学学院);Research Ireland Insight Centre and ADAPT Centre, University of Galway, Ireland(Galway大学爱尔兰研究洞察中心和ADAPT研究中心)
Cross-Lingual Training with Dense Retrieval for Document Retrieval
David R. Cheriton School of Computer Science, University of Waterloo(滑铁卢大学戴维·R·切里顿计算机科学学院);Department of Computer Science and Engineering, Penn State University(宾夕法尼亚州立大学计算机科学与工程系)
Ubiquitous Knowledge Processing Lab (UKP Lab)(普适知识处理实验室(UKP实验室));Department of Computer Science(计算机科学系);Hessian Center for AI (hessian.AI)(黑森人工智能中心(hessian.AI));Technical University of Darmstadt(达姆施塔特工业大学);cohere(Cohere公司)
ConTextual Masked Auto-Encoder for Dense Passage Retrieval
Institute of Information Engineering, Chinese Academy of Sciences(中国科学院信息工程研究所);School of Cyber Security, University of Chinese Academy of Sciences(中国科学院大学网络空间安全学院);Kuaishou Technology(快手科技)
Retrieval Oriented Masking Pre-training Language Model for Dense Passage Retrieval
Alibaba Group(阿里巴巴集团)
预训练的语言模型(PTM)已被证明产生强大的文本表示密集段落检索任务。Masked Language Modeling(MLM)是预训练过程的一个主要子任务。然而,我们发现,传统的随机掩蔽策略倾向于选择大量的令牌,对段落检索任务的影响有限(例如,停止词和标点符号)。通过注意到术语重要性权重可以为段落检索提供有价值的信息,我们在此提出了替代的面向检索的掩蔽(称为ROM)策略,其中更重要的标记将具有更高的被掩蔽的概率,以捕获这种简单而重要的信息,以促进语言模型预训练过程。值得注意的是,所提出的新的令牌掩蔽方法不会改变原始PTM的架构和学习目标。我们的实验验证了所提出的ROM使术语重要性信息,以帮助语言模型预训练,从而实现更好的性能多通道检索基准。
Topic-DPR: Topic-based Prompts for Dense Passage Retrieval
South China Normal University(华南师范大学);The Hong Kong University of Science and Technology(香港科技大学);The Hong Kong University of Science and Technology (Guangzhou)(香港科技大学(广州))
Improve Dense Passage Retrieval with Entailment Tuning
The Hong Kong University of Science and Technology (Guangzhou)(香港科技大学(广州));The Hong Kong University of Science and Technology(香港科技大学);Thrust of Artificial Intelligence(人工智能分支(或:人工智能 thrust,按原文直译))
Tsinghua University(清华大学);Quan Cheng Laboratory(泉城实验室);Renmin University of China(中国人民大学);Gaoling School of Artificial Intelligence, Renmin University of China(中国人民大学高瓴人工智能学院);Zhongguancun Laboratory(中关村实验室);Xiaohongshu Inc(小红书科技有限公司)