arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2409.02727cs.CLcs.IR
收藏

池化与注意力:基于LLM的嵌入模型的有效设计是什么?

Pooling And Attention: What Are Effective Designs For LLM-Based Embedding Models?

The Hong Kong University of Science and Technology(香港科技大学)

Yixuan Tang, Yi Yang

AI总结:

本文通过统一数据和基础模型的大规模实验,系统比较不同池化与注意力策略对LLM嵌入模型的影响,并提出多层可训练池化方法,在文本相似度和检索任务中取得统计显著提升。

摘要:

大型语言模型(LLMs)在生成任务中的显著进展催生了大量探索基于LLM的嵌入模型的工作。尽管这些模型采用不同的池化和注意力策略,已在公开嵌入基准上取得了最先进的性能,但关于什么构成基于LLM的嵌入模型的有效设计这一问题仍然存在。然而,这些模型通常在训练时使用不同的数据集、不同的LLM基础模型或不同的训练设置。此外,在公开嵌入基准上的评估往往未能报告统计显著性,这使得判断哪些设计真正对最终性能有所贡献变得困难。这给寻求基于LLM的嵌入模型最优训练方案的实践者带来了复杂性。在本研究中,我们通过训练一系列基于LLM的嵌入模型进行了大规模实验,这些模型使用相同的训练数据和基础模型,但在池化和注意力策略上有所不同。结果表明,不存在放之四海而皆准的解决方案:虽然双向注意力和额外的可训练池化层在文本相似度和信息检索任务中表现更优,但在聚类和分类任务中,它们并未显著超越更简单的设计,如EOS-last token池化和默认因果注意力。此外,我们提出了一种新的池化策略,即多层可训练池化(Multi-Layers Trainable Pooling),该策略使用交叉注意力网络转换所有隐藏层的输出,而不仅仅是最后一层。与现有池化方法相比,该方法在文本相似度和检索任务中被证明具有统计上的优越性。总体而言,本文揭示了基于LLM的嵌入模型的有效训练策略。

英文摘要:

The significant advancements of Large Language Models (LLMs) in generative tasks have led to a growing body of work exploring LLM-based embedding models. While these models, employing different pooling and attention strategies, have achieved state-of-the-art performance on public embedding benchmarks, questions still arise about what constitutes an effective design for LLM-based embedding models. However, these models are often trained on different datasets, using different LLM base models or training settings. Moreover, evaluations on public embedding benchmarks often fail to report statistical significance, making it difficult to determine which designs truly contribute to final performance. This complicates the process for practitioners seeking optimal training recipes for LLM-based embedding models. In this study, we conduct a large-scale experiment by training a series of LLM-based embedding models using the same training data and base model but differing in their pooling and attention strategies. The results show that there is no one-size-fits-all solution: while bidirectional attention and an additional trainable pooling layer outperform in text similarity and information retrieval tasks, they do not significantly surpass simpler designs like EOS-last token pooling and default causal attention in clustering and classification tasks. Furthermore, we propose a new pooling strategy, Multi-Layers Trainable Pooling, which transforms the outputs of all hidden layers, rather than just the last layer, using a cross-attention network. This method proves to be statistically superior in text similarity and retrieval tasks compared to existing pooling methods. Overall, this paper sheds light on effective training strategies for LLM-based embedding models.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑