Dense retrievers utilize pre-trained backbone language models (e.g., BERT, LLaMA) that are fine-tuned via contrastive learning to perform the task of encoding text into sense representations that can be then compared via a shallow similarity operation, e.g. inner product. Recent research has questioned the role of fine-tuning vs. that of pre-training within dense retrievers, specifically arguing that retrieval knowledge is primarily gained during pre-training, meaning knowledge not acquired during pre-training cannot be sub-sequentially acquired via fine-tuning. We revisit this idea here as the claim was only studied in the context of a BERT-based encoder using DPR as representative dense retriever. We extend the previous analysis by testing other representation approaches (comparing the use of CLS tokens with that of mean pooling), backbone architectures (encoder-only BERT vs. decoder-only LLaMA), and additional datasets (MSMARCO in addition to Natural Questions). Our study confirms that in DPR tuning, pre-trained knowledge underpins retrieval performance, with fine-tuning primarily adjusting neuron activation rather than reorganizing knowledge. However, this pattern does not hold universally, such as in mean-pooled (Contriever) and decoder-based (LLaMA) models. We ensure full reproducibility and make our implementation publicly available at https://github.com/ielab/DenseRetriever-Knowledge-Acquisition.
Where Does the Performance Improvement Come From? -- A Reproducibility Concern about Image-Text Retrieval
Harbin Institute of Technology, Shenzhen(哈尔滨工业大学(深圳));China University of Petroleum (East China)(中国石油大学(华东));JD Explore Academy(京东探索研究院);Peng Cheng Laboratory(鹏城实验室)
A Reproducibility Study of LLM-Based Query Reformulation
University of Waterloo(滑铁卢大学);Mila – Quebec AI Institute(魁北克AI研究所);Toronto Metropolitan University(多伦多 Metropolitan 大学);University of Toronto(多伦多大学);University of California, Berkeley(加州大学伯克利分校)
Acquisition of interpretable domain information during brain MR image harmonization for content-based image retrieval
Dept. of Applied Informatics, Graduate School of Science and Engineering, Hosei University(应用信息科,理学部,法政大学);Dept. of Radiology and Radiological Science, Johns Hopkins Medicine(放射科和放射科学系,约翰霍普金斯医学中心)