探究密集检索中的混合专家模型
Investigating Mixture of Experts in Dense Retrieval
AI总结：
本研究在密集检索模型最终Transformer层后集成单个MoE块（SB-MoE），实验表明其对小参数模型（如TinyBERT）在四个基准上均优于标准微调，而大参数模型需更多训练样本。
摘要：
尽管密集检索模型（DRMs）推动了信息检索（IR）的发展，但这些神经模型的一个局限性在于其泛化能力和鲁棒性较窄。为了解决这一问题，可以利用混合专家（MoE）架构。虽然以往的IR研究已将MoE架构整合到DRMs的Transformer层中，但我们的工作研究了一种在最终Transformer层输出之后集成单个MoE块（SB-MoE）的架构。我们的实证评估考察了SB-MoE在检索效果方面与标准微调的比较。具体而言，我们在四个基准数据集上，分别在有和没有添加MoE块的情况下，对三种DRMs（TinyBERT、BERT和Contriever）进行了微调。此外，由于MoE的参数（即专家数量）不同会导致性能变化，我们进行了额外实验以进一步探究这一方面。研究结果表明，SB-MoE尤其对于参数数量较少的DRMs（即TinyBERT）具有有效性，因为它在所有四个基准上始终优于微调后的基础模型。对于参数数量较多的DRMs（即BERT和Contriever），SB-MoE需要更大数量的训练样本才能获得更好的检索性能。
英文摘要：
While Dense Retrieval Models (DRMs) have advanced Information Retrieval (IR), one limitation of these neural models is their narrow generalizability and robustness. To cope with this issue, one can leverage the Mixture-of-Experts (MoE) architecture. While previous IR studies have incorporated MoE architectures within the Transformer layers of DRMs, our work investigates an architecture that integrates a single MoE block (SB-MoE) after the output of the final Transformer layer. Our empirical evaluation investigates how SB-MoE compares, in terms of retrieval effectiveness, to standard fine-tuning. In detail, we fine-tune three DRMs (TinyBERT, BERT, and Contriever) across four benchmark collections with and without adding the MoE block. Moreover, since MoE showcases performance variations with respect to its parameters (i.e., the number of experts), we conduct additional experiments to investigate this aspect further. The findings show the effectiveness of SB-MoE especially for DRMs with a low number of parameters (i.e., TinyBERT), as it consistently outperforms the fine-tuned underlying model on all four benchmarks. For DRMs with a higher number of parameters (i.e., BERT and Contriever), SB-MoE requires larger numbers of training samples to yield better retrieval performance.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
