The significant advancements of Large Language Models (LLMs) in generative tasks have led to a growing body of work exploring LLM-based embedding models. While these models, employing different pooling and attention strategies, have achieved state-of-the-art performance on public embedding benchmarks, questions still arise about what constitutes an effective design for LLM-based embedding models. However, these models are often trained on different datasets, using different LLM base models or training settings. Moreover, evaluations on public embedding benchmarks often fail to report statistical significance, making it difficult to determine which designs truly contribute to final performance. This complicates the process for practitioners seeking optimal training recipes for LLM-based embedding models. In this study, we conduct a large-scale experiment by training a series of LLM-based embedding models using the same training data and base model but differing in their pooling and attention strategies. The results show that there is no one-size-fits-all solution: while bidirectional attention and an additional trainable pooling layer outperform in text similarity and information retrieval tasks, they do not significantly surpass simpler designs like EOS-last token pooling and default causal attention in clustering and classification tasks. Furthermore, we propose a new pooling strategy, Multi-Layers Trainable Pooling, which transforms the outputs of all hidden layers, rather than just the last layer, using a cross-attention network. This method proves to be statistically superior in text similarity and retrieval tasks compared to existing pooling methods. Overall, this paper sheds light on effective training strategies for LLM-based embedding models.
Selecting What Matters: Semantic Compression-Guided Selective Pooling for Long-Context Embeddings
State Key Laboratory for Novel Software Technology, Nanjing University(南京大学计算机软件新技术国家重点实验室);National University of Singapore(新加坡国立大学);Nanjing University of Posts and Telecommunications(南京邮电大学)
Lessons Learned on Information Retrieval in Electronic Health Records: A Comparison of Embedding Models and Pooling Strategies
University of Wisconsin-Madison(威斯康星大学麦迪逊分校);Boston Children’s Hospital Harvard Medical School(波士顿儿童医院哈佛医学院);University of Colorado Anschutz(科罗拉多大学安舒茨分校);Loyola University Chicago(芝加哥洛约拉大学)
Generalized Stock Price Prediction for Multiple Stocks Combined with News Fusion
Graduate Institute of Electrical Engineering, National Taiwan University(国立台湾大学电子工程研究所);Institute of Information Science, Academia Sinica(学术院信息科学研究所);United Link Co., Ltd.(联合链接公司);University of Washington(华盛顿大学);National Tsing Hua University(国立清华大学)
What Should Embeddings Embed? Autoregressive Models Represent Latent Generating Distributions
Department of Computer Science(计算机科学系);Princeton University(普林斯顿大学);Stanford University(斯坦福大学);Department of Linguistics and Wu Tsai Institute(语言学系和吴泰教授研究所);Yale University(耶鲁大学);Anthropic(Anthropic公司);Departments of Psychology and Computer Science(心理学系和计算机科学系)
Comparing representations of long clinical texts for the task of patient note-identification
Inria(法国国家信息与自动化研究所);Inserm(法国国家健康与医学研究院);Data Science Platform, Imagine Institute(伊马金研究所数据科学平台);Hôpital Universitaire Necker-Enfants Malades(内克尔-恩范茨-马拉德大学医院);Assistance Publique - Hôpitaux de Paris (APHP)(巴黎公共医疗援助医院集团);Université de Lorraine(洛林大学);CNRS(法国国家科学研究中心);LORIA(洛林计算机科学与应用实验室);INESC-ID(里斯本大学学院计算机与系统工程研究中心);Instituto Superior Técnico(里斯本高等技术学院);Universidade de Lisboa(里斯本大学)
Enhancing Hyperspace Analogue to Language (HAL) Representations via Attention-Based Pooling for Text Classification
Undergraduate Students, Department of Computer and Automatic Control Engineering, Faculty of Mechanical and Electrical Engineering, Lattakia University(本科生,计算机与自动控制工程系,机械与电气工程学院,拉塔基亚大学)
Theoretical foundations and limits of word embeddings: what types of meaning can they capture?
University of California, Los Angeles(加利福尼亚大学洛杉矶分校)
意义的测量是文化社会学的一个核心问题,而词嵌入可以提供强大的新工具来做到这一点。但就像任何工具一样,它们建立在理论假设之上并发挥作用。在本文中,我理论化的方式,其中词嵌入模型的三个核心前提的结构语言学理论的意义:意义是关系,连贯的,可以作为一个静态系统进行分析。在某些方面,词嵌入方法容易受到这些前提的相同的、持久的批评。在其他方面,他们为这些批评提供了新的解决方案。更广泛地说,用词嵌入形式化意义的研究为澄清文化社会学中的核心概念和争论提供了理论机会,例如意义的连贯性。正如网络分析明确了曾经模糊的社会关系概念(Borgatti et al. 2009),用嵌入方法形式化意义可以推动我们明确和重新想象意义本身。