Mamba Retriever：利用 Mamba 实现有效且高效的密集检索
Mamba Retriever: Utilizing Mamba for Effective and Efficient Dense Retrieval
AI总结：
本文提出 Mamba Retriever，利用 Mamba 架构作为密集检索的编码器，在短文本和长文本数据集上微调，实验证明其在有效性和推理速度上均优于或等同于 Transformer 模型，尤其适用于长文本检索。
摘要：
在信息检索（IR）领域，密集检索（DR）模型使用深度学习技术将查询和段落编码到嵌入空间中以计算它们的语义关系。对于 DR 模型而言，平衡效率和有效性非常重要。预训练语言模型（PLMs），特别是基于 Transformer 的 PLMs，已被证明是 DR 模型的有效编码器。然而，基于 Transformer 的 PLM 中的自注意力组件导致计算复杂度随序列长度呈二次方增长，因此在长文本检索中表现出较慢的推理速度。一些最近提出的非 Transformer PLMs，特别是 Mamba 架构的 PLMs，不仅在生成式语言任务上展示了与基于 Transformer 的 PLMs 相当的有效性，而且由于序列长度上的线性时间缩放而表现出更好的效率。本文实现了 Mamba Retriever 以探索 Mamba 是否能作为 IR 任务中 DR 模型的有效且高效的编码器。我们在经典的短文本 MS MARCO 段落排序数据集和长文本 LoCoV0 数据集上对 Mamba Retriever 进行了微调。实验结果表明：（1）在 MS MARCO 段落排序数据集和 BEIR 上，Mamba Retriever 与基于 Transformer 的检索模型相比取得了相当或更好的有效性，并且有效性随 Mamba 模型规模的增大而提升；（2）在长文本 LoCoV0 数据集上，Mamba Retriever 在检索任务上微调后可以扩展到比其预训练长度更长的文本长度，并且与其他长文本检索模型相比具有相当或更好的有效性；（3）Mamba Retriever 在长文本检索中具有卓越的推理速度。总之，Mamba Retriever 既有效又高效，使其成为一个实用的模型，特别是对于长文本检索。
英文摘要：
In the information retrieval (IR) area, dense retrieval (DR) models use deep learning techniques to encode queries and passages into embedding space to compute their semantic relations. It is important for DR models to balance both efficiency and effectiveness. Pre-trained language models (PLMs), especially Transformer-based PLMs, have been proven to be effective encoders of DR models. However, the self-attention component in Transformer-based PLM results in a computational complexity that grows quadratically with sequence length, and thus exhibits a slow inference speed for long-text retrieval. Some recently proposed non-Transformer PLMs, especially the Mamba architecture PLMs, have demonstrated not only comparable effectiveness to Transformer-based PLMs on generative language tasks but also better efficiency due to linear time scaling in sequence length. This paper implements the Mamba Retriever to explore whether Mamba can serve as an effective and efficient encoder of DR model for IR tasks. We fine-tune the Mamba Retriever on the classic short-text MS MARCO passage ranking dataset and the long-text LoCoV0 dataset. Experimental results show that (1) on the MS MARCO passage ranking dataset and BEIR, the Mamba Retriever achieves comparable or better effectiveness compared to Transformer-based retrieval models, and the effectiveness grows with the size of the Mamba model; (2) on the long-text LoCoV0 dataset, the Mamba Retriever can extend to longer text length than its pre-trained length after fine-tuning on retrieval task, and it has comparable or better effectiveness compared to other long-text retrieval models; (3) the Mamba Retriever has superior inference speed for long-text retrieval. In conclusion, Mamba Retriever is both effective and efficient, making it a practical model, especially for long-text retrieval.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
