arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2409.04701cs.CLcs.IR
收藏

后期分块:利用长上下文嵌入模型的上下文分块嵌入

Late Chunking: Contextual Chunk Embeddings Using Long-Context Embedding Models

Jina AI GmbH(Jina AI公司);Weaviate B.V.(Weaviate公司)

Michael Günther, Isabelle Mohr, Daniel James Williams, Bo Wang, Han Xiao

AI总结:

本文提出后期分块方法,在长上下文嵌入模型编码全部 token 后再进行分块和均值池化,使分块嵌入保留完整上下文信息,从而提升检索效果,并进一步提出专用微调方案。

摘要:

许多使用场景需要检索较小的文本片段,而基于稠密向量的检索系统通常在文本片段较短时表现更好,因为语义在嵌入中不太可能被过度压缩。因此,实践者通常将文本文档分割成较小的块并分别进行编码。然而,以这种方式创建的分块嵌入可能会丢失周围分块的上下文信息,从而导致次优的表示。在本文中,我们引入了一种名为后期分块(late chunking)的新方法,该方法利用长上下文嵌入模型首先对长文本的所有 token 进行嵌入,分块操作在 transformer 模型之后、均值池化之前进行——因此其名称中包含“后期”一词。由此产生的分块嵌入捕获了完整的上下文信息,从而在各种检索任务中取得了更优的结果。该方法具有足够的通用性,可应用于广泛的长上下文嵌入模型,并且无需额外训练即可工作。为了进一步提高后期分块的有效性,我们提出了一种专门的嵌入模型微调方法。

英文摘要:

Many use cases require retrieving smaller portions of text, and dense vector-based retrieval systems often perform better with shorter text segments, as the semantics are less likely to be over-compressed in the embeddings. Consequently, practitioners often split text documents into smaller chunks and encode them separately. However, chunk embeddings created in this way can lose contextual information from surrounding chunks, resulting in sub-optimal representations. In this paper, we introduce a novel method called late chunking, which leverages long context embedding models to first embed all tokens of the long text, with chunking applied after the transformer model and just before mean pooling - hence the term late in its naming. The resulting chunk embeddings capture the full contextual information, leading to superior results across various retrieval tasks. The method is generic enough to be applied to a wide range of long-context embedding models and works without additional training. To further increase the effectiveness of late chunking, we propose a dedicated fine-tuning approach for embedding models.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑