使大语言模型成为高效密集检索器
Making Large Language Models Efficient Dense Retrievers
AI总结：
EffiR通过粗到细的MLP压缩策略和检索特定微调，使大语言模型在保持性能的同时显著减少模型大小和推理成本。
摘要：
最近的研究表明，直接微调大语言模型（LLMs）进行密集检索可以取得优异的性能，但它们庞大的参数数量使它们在计算上效率低下。尽管先前的研究揭示了生成任务中LLMs的显著层数冗余，但这些模型适应于检索任务时是否存在类似的冗余仍不清楚，因为检索任务需要将整个序列编码为固定表示，而不是迭代生成标记。为此，我们对LLM基于密集检索器的层冗余进行了全面分析。我们发现，与生成设置不同，MLP层在很大程度上可以被压缩，而注意力层对于语义聚合仍然是关键的。基于这一见解，我们提出了EffiR，一种开发高效检索器的框架，通过粗到细的策略（粗粒度深度减少后接细粒度宽度减少）进行大规模MLP压缩，并结合特定于检索的微调。在多样化的BEIR数据集和LLM基础架构上，EffiR在保持全尺寸模型性能的同时，实现了模型大小和推理成本的显著减少。
英文摘要：
Recent work has shown that directly fine-tuning large language models (LLMs) for dense retrieval yields strong performance, but their substantial parameter counts make them computationally inefficient. While prior studies have revealed significant layer redundancy in LLMs for generative tasks, it remains unclear whether similar redundancy exists when these models are adapted for retrieval tasks, which require encoding entire sequences into fixed representations rather than generating tokens iteratively. To this end, we conduct a comprehensive analysis of layer redundancy in LLM-based dense retrievers. We find that, in contrast to generative settings, MLP layers are substantially more prunable, while attention layers remain critical for semantic aggregation. Building on this insight, we propose EffiR, a framework for developing efficient retrievers that performs large-scale MLP compression through a coarse-to-fine strategy (coarse-grained depth reduction followed by fine-grained width reduction), combined with retrieval-specific fine-tuning. Across diverse BEIR datasets and LLM backbones, EffiR achieves substantial reductions in model size and inference cost while preserving the performance of full-size models.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
