利用查询似然建模释放大型语言模型在密集检索中的能力
Unleashing the Power of LLMs in Dense Retrieval with Query Likelihood Modeling
AI总结：
该研究针对LLM在密集检索中全局信息建模不足的问题，提出含注意力块和文档损坏组件的LLM-QL模型，通过查询似然最大化辅助任务增强检索器主干，在MS MARCO和BEIR数据集上优于其他LLM-based检索器。
摘要：
密集检索是信息检索（IR）中的一项关键任务，是重排序、增强生成等下游任务的基础。近年来，大型语言模型（LLM）展现出出色的语义理解能力，受到密集检索研究者的关注。但作为解码器式生成模型的LLM虽擅长语言生成，却因缺乏对后续token的关注，在全局信息建模上存在不足。受IR中经典的基于词的语言建模方法——查询似然（QL）模型启发，我们旨在通过QL最大化利用LLM的生成优势：不将QL估计用于文档排序，而是提出QL最大化的辅助任务，以增强密集检索器的主干网络，用于后续的对比学习。我们提出的模型LLM-QL包含两个关键组件：注意力块（Attention Block, AB）和文档损坏（Document Corruption, DC）。AB会阻止预测token关注文档结束token之前的文档token；DC则在预测时通过掩码文档的部分token来损坏文档。在域内数据集MS MARCO和域外数据集BEIR上的评估显示，LLM-QL优于其他基于LLM的检索器；进一步的综合分析也验证了LLM-QL及其组件的有效性。
英文摘要：
Dense retrieval is a crucial task in Information Retrieval (IR), serving as the basis for downstream tasks such as re-ranking and augmenting generation. Recently, large language models (LLMs) have demonstrated impressive semantic understanding capabilities, making them attractive to researchers focusing on dense retrieval. While LLMs, as decoder-style generative models, excel in language generation, they often fall short in modeling global information due to a lack of attention to subsequent tokens. Drawing inspiration from the classical word-based language modeling approach for IR, specifically the query likelihood (QL) model, we aim to leverage the generative strengths of LLMs through QL maximization. Rather than employing QL estimation for document ranking, we propose an auxiliary task of QL maximization to enhance the backbone for subsequent contrastive learning of the retriever. We introduce our model, LLM-QL, which incorporates two key components: Attention Block (AB) and Document Corruption (DC). AB blocks the attention of predictive tokens to the document tokens before the document's ending token, while DC corrupts a document by masking a portion of its tokens during prediction. Evaluations on the in-domain (MS MARCO) and out-of-domain dataset (BEIR) indicate LLM-QL's superiority over other LLM-based retrievers. Furthermore, comprehensive analyses also validate the efficacy of LLM-QL and its components.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
