arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2505.07166cs.IRcs.CL
收藏

预训练与微调:关于稠密检索知识获取的可复现性研究

Pre-training vs. Fine-tuning: A Reproducibility Study on Dense Retrieval Knowledge Acquisition

The University of Queensland(昆士兰大学)

Zheng Yao, Shuai Wang, Guido Zuccon

AI总结:

本文复现并扩展稠密检索知识获取研究,发现 DPR 中预训练主导知识、微调主要改变激活,但该结论不适用于 Contriever 和 LLaMA。

摘要:

稠密检索器使用预训练骨干语言模型(例如 BERT、LLaMA),并通过对比学习进行微调,以完成将文本编码为语义表示的任务,这些表示随后可通过浅层相似度操作进行比较,例如内积。近期研究质疑了稠密检索器中微调相对于预训练的作用,明确主张检索知识主要是在预训练期间获得的,这意味着预训练期间未获得的知识无法在后续通过微调获得。我们在此重新审视这一观点,因为该主张此前只在基于 BERT 的编码器、并以 DPR 作为代表性稠密检索器的情境下得到研究。我们扩展了先前的分析,测试了其他表示方法(比较 CLS token 与均值池化的使用)、骨干架构(仅编码器 BERT 与仅解码器 LLaMA)以及更多数据集(在 Natural Questions 之外加入 MSMARCO)。我们的研究证实,在 DPR 调优中,预训练知识支撑着检索性能,微调主要调整神经元激活,而不是重组知识。然而,这一模式并不普遍成立,例如在均值池化的 Contriever 和基于解码器的 LLaMA 模型中即是如此。我们确保完全可复现,并在 https://github.com/ielab/DenseRetriever-Knowledge-Acquisition 公开实现。

英文摘要:

Dense retrievers utilize pre-trained backbone language models (e.g., BERT, LLaMA) that are fine-tuned via contrastive learning to perform the task of encoding text into sense representations that can be then compared via a shallow similarity operation, e.g. inner product. Recent research has questioned the role of fine-tuning vs. that of pre-training within dense retrievers, specifically arguing that retrieval knowledge is primarily gained during pre-training, meaning knowledge not acquired during pre-training cannot be sub-sequentially acquired via fine-tuning. We revisit this idea here as the claim was only studied in the context of a BERT-based encoder using DPR as representative dense retriever. We extend the previous analysis by testing other representation approaches (comparing the use of CLS tokens with that of mean pooling), backbone architectures (encoder-only BERT vs. decoder-only LLaMA), and additional datasets (MSMARCO in addition to Natural Questions). Our study confirms that in DPR tuning, pre-trained knowledge underpins retrieval performance, with fine-tuning primarily adjusting neuron activation rather than reorganizing knowledge. However, this pattern does not hold universally, such as in mean-pooled (Contriever) and decoder-based (LLaMA) models. We ensure full reproducibility and make our implementation publicly available at https://github.com/ielab/DenseRetriever-Knowledge-Acquisition.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑