arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2210.01371cs.IRcs.CL
收藏

轻量级混合检索器的效率与泛化研究

A Study on the Efficiency and Generalization of Light Hybrid Retrievers

Arizona State University(亚利桑那州立大学);Meta Reality Lab(Meta现实实验室)

Man Luo, Shashank Jain, Anchit Gupta, Arash Einolghozati, Barlas Oguz, Debojeet Chatterjee, Xilun Chen, Chitta Baral, Peyman Heidari

AI总结:

本文提出轻量级混合检索器 Hybrid-LITE,通过结合 BM25 与基于 DrBoost 知识蒸馏训练的 LITE,在节省 13 倍内存的同时保持 98.0% 性能,并展现出更优的域外泛化能力。

摘要:

混合检索器可以同时利用稀疏检索器和密集检索器的优势。以往的混合检索器依赖于索引繁重的密集检索器。在本研究中,我们探讨“是否可以在不牺牲性能的情况下减少混合检索器的索引内存”?受此问题驱动,我们采用了一种索引高效的密集检索器(即 DrBoost),并引入了一种进一步减少 DrBoost 内存的 LITE retriever。LITE 在对比学习和来自 DrBoost 的知识蒸馏上进行联合训练。然后,我们将稀疏检索器 BM25 与 LITE 或 DrBoost 集成,形成轻量级混合检索器。我们的 Hybrid-LITE retriever 节省了 13 倍的内存,同时保持了 BM25 和 DPR 混合检索器 98.0% 的性能。此外,我们在域外数据集和一组对抗性攻击数据集上研究了轻量级混合检索器的泛化能力。实验表明,轻量级混合检索器比单独的稀疏和密集检索器取得了更好的泛化性能。尽管如此,我们的分析表明,检索器的鲁棒性仍有很大的改进空间,这暗示了一个新的研究方向。

英文摘要:

Hybrid retrievers can take advantage of both sparse and dense retrievers. Previous hybrid retrievers leverage indexing-heavy dense retrievers. In this work, we study "Is it possible to reduce the indexing memory of hybrid retrievers without sacrificing performance"? Driven by this question, we leverage an indexing-efficient dense retriever (i.e. DrBoost) and introduce a LITE retriever that further reduces the memory of DrBoost. LITE is jointly trained on contrastive learning and knowledge distillation from DrBoost. Then, we integrate BM25, a sparse retriever, with either LITE or DrBoost to form light hybrid retrievers. Our Hybrid-LITE retriever saves 13X memory while maintaining 98.0% performance of the hybrid retriever of BM25 and DPR. In addition, we study the generalization capacity of our light hybrid retrievers on out-of-domain dataset and a set of adversarial attacks datasets. Experiments showcase that light hybrid retrievers achieve better generalization performance than individual sparse and dense retrievers. Nevertheless, our analysis shows that there is a large room to improve the robustness of retrievers, suggesting a new research direction.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑