arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2504.18151cs.IR
收藏

利用解码器架构进行学习型稀疏检索

Leveraging Decoder Architectures for Learned Sparse Retrieval

Jingfen Qiao, Thong Nguyen, Evangelos Kanoulas, Andrew Yates

AI总结:

本研究探索了学习型稀疏检索在仅编码器、仅解码器和编码器-解码器架构中的表现,发现编码器-解码器架构配合多令牌解码效果最佳,而仅解码器模型在参数规模增大时具有超越潜力。

摘要:

学习型稀疏检索(LSR)传统上专注于小规模的仅编码器Transformer架构。随着大规模预训练语言模型的出现,它们在不同基于Transformer的架构(包括仅编码器、仅解码器和编码器-解码器模型)中为检索任务生成稀疏表示的能力在很大程度上仍未得到探索。本研究调查了LSR在这些架构中的有效性,探索了各种稀疏表示头(sparse representation heads)和模型规模。我们的结果突出了在零样本设置中使用大型语言模型创建有效稀疏表示的局限性,指出了诸如不恰当的术语扩展以及因缺乏扩展而导致性能下降等挑战。我们发现,采用多令牌解码(multi-tokens decoding)方法的编码器-解码器架构在三种骨干网络中表现最佳。虽然仅解码器模型的表现不如仅编码器模型,但当扩展到大量参数时,它显示出超越的潜力。

英文摘要:

Learned Sparse Retrieval (LSR) has traditionally focused on small-scale encoder-only transformer architectures. With the advent of large-scale pre-trained language models, their capability to generate sparse representations for retrieval tasks across different transformer-based architectures, including encoder-only, decoder-only, and encoder-decoder models, remains largely unexplored. This study investigates the effectiveness of LSR across these architectures, exploring various sparse representation heads and model scales. Our results highlight the limitations of using large language models to create effective sparse representations in zero-shot settings, identifying challenges such as inappropriate term expansions and reduced performance due to the lack of expansion. We find that the encoder-decoder architecture with multi-tokens decoding approach achieves the best performance among the three backbones. While the decoder-only model performs worse than the encoder-only model, it demonstrates the potential to outperform when scaled to a high number of parameters.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑