arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2311.02834cs.IR
收藏

CAME:面向第一阶段检索的混合专家模型竞争性学习方法

CAME: Competitively Learning a Mixture-of-Experts Model for First-stage Retrieval

Yinqiong Cai, Yixing Fan, Keping Bi, Jiafeng Guo, Wei Chen, Ruqing Zhang, Xueqi Cheng

AI总结:

针对现有集成检索模型未适配单一样本匹配模式、无法充分发挥不同模型能力的问题,本文提出带竞争性学习机制的混合专家模型CAME,通过两阶段训练让专家发展专长,在三个基准数据集上显著优于最优基线。

摘要:

第一阶段检索旨在从海量文档集合中高效且有效地检索出候选文档子集。由于查询与相关文档之间可能存在多种匹配模式,已有研究尝试结合多种检索模型以尽可能多地召回相关结果。但无论是独立训练还是联合训练构建的集成模型,在训练过程中都未关注哪个组件模型更适配单个样本,因此无法充分发挥不同类型检索模型在识别多样化相关性模式方面的能力。受此观察启发,本文提出一种由代表性匹配专家组成的混合专家(Mixture-of-Experts, MoE)模型,以及一种新颖的竞争性学习机制,让专家在训练过程中发展并强化自身专长。具体而言,我们的MoE模型共享底层网络以学习通用语义表示,采用不同结构的上层网络表征不同类型的检索专家。该竞争性学习机制包含两个阶段:(1)标准化学习阶段,对所有专家进行同等训练,培养其执行相关性匹配的基础能力;(2)专业化学习阶段,专家在每个训练样本上相互竞争,并根据自身表现获得奖励与参数更新,以强化其在特定类型样本上的专长。在三个检索基准数据集上的实验结果表明,我们的方法显著优于当前最优基线模型。

英文摘要:

The first-stage retrieval aims to retrieve a subset of candidate documents from a huge collection both effectively and efficiently. Since various matching patterns can exist between queries and relevant documents, previous work tries to combine multiple retrieval models to find as many relevant results as possible. The constructed ensembles, whether learned independently or jointly, do not care which component model is more suitable to an instance during training. Thus, they cannot fully exploit the capabilities of different types of retrieval models in identifying diverse relevance patterns. Motivated by this observation, in this paper, we propose a Mixture-of-Experts (MoE) model consisting of representative matching experts and a novel competitive learning mechanism to let the experts develop and enhance their expertise during training. Specifically, our MoE model shares the bottom layers to learn common semantic representations and uses differently structured upper layers to represent various types of retrieval experts. Our competitive learning mechanism has two stages: (1) a standardized learning stage to train the experts equally to develop their capabilities to conduct relevance matching; (2) a specialized learning stage where the experts compete with each other on every training instance and get rewards and updates according to their performance to enhance their expertise on certain types of samples. Experimental results on three retrieval benchmark datasets show that our method significantly outperforms the state-of-the-art baselines.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑