LLMs 与隔离内核:轻量、无学习的二进制嵌入用于快速检索
LLMs Meet Isolation Kernel: Lightweight, Learning-free Binary Embeddings for Fast Retrieval
AI总结:
本文提出无学习的隔离内核嵌入(IKE),将LLM嵌入转换为二进制嵌入,实现低内存和快速检索,实验显示其检索速度提升16.7倍,内存使用降低16倍,同时保持相似精度。
摘要:
大型语言模型(LLMs)最近在文本表示方面取得了显著进展。然而,其嵌入通常是高维的,导致存储和检索开销巨大。尽管最近的方法如Matryoshka Representation Learning(MRL)和Contrastive Sparse Representation(CSR)在一定程度上缓解了这些问题,但仍存在检索精度下降的问题。本文提出隔离内核嵌入或IKE,一种无学习方法,通过隔离内核(IK)将LLM嵌入转换为二进制嵌入。轻量且基于二进制编码,IKE提供了低内存足迹和快速位计算,降低了检索延迟。在多个文本检索数据集上的实验表明,IKE相比原始LLM嵌入,检索速度提高16.7倍,内存使用降低16倍,同时保持相似精度。理论上,我们证明IKE有效是因为它满足了有效二进制哈希的四个基本标准,而其他方法不具备。与CSR相比,IKE在检索效率和效果上始终更优。IKE也有效果地与基于图的索引结合,展示了其在近邻搜索设置中平衡精度和延迟方面的优越性,相比其他压缩技术。
英文摘要:
Large language models (LLMs) have recently enabled remarkable progress in text representation. However, their embeddings are typically high-dimensional, leading to substantial storage and retrieval overhead. Although recent approaches such as Matryoshka Representation Learning (MRL) and Contrastive Sparse Representation (CSR) alleviate these issues to some extent, they still suffer from retrieval accuracy degradation. This paper proposes Isolation Kernel Embedding or IKE, a learning-free method that transforms an LLM embedding into a binary embedding using Isolation Kernel (IK). Lightweight and based on binary encoding, IKE offers a low memory footprint and fast bitwise computation, lowering retrieval latency. Experiments on multiple text retrieval datasets demonstrate that IKE offers up to 16.7x faster retrieval and 16x lower memory usage than the original LLM embeddings, while maintaining comparable accuracy. Theoretically, we show that IKE works because it satisfies four essential criteria for effective binary hashing that other methods do not possess. Compared to CSR, IKE consistently achieves better retrieval efficiency and effectiveness. IKE also works effectively with graph-based indexing, demonstrating its superiority in balancing accuracy and latency compared to alternative compression techniques in the approximate nearest neighbor (ANN) search setting.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。