基于Voronoi单元的原理化令牌修剪方法:用于晚期交互检索模型
A Voronoi Cell Formulation for Principled Token Pruning in Late-Interaction Retrieval Models
AI总结:
本文提出基于超空间几何的Voronoi单元方法,用于原理化修剪晚期交互检索模型中的令牌嵌入,通过保留检索质量的同时减少索引存储开销。
摘要:
晚期交互模型如ColBERT在各种检索任务中表现优异,但需要为每个文档令牌存储密集嵌入,导致显著的索引存储开销。以往工作通过统计和经验措施尝试修剪低重要性令牌嵌入,但往往缺乏正式理论基础或效果不佳。为解决这些问题,我们引入基于超空间几何的框架,将令牌修剪视为嵌入空间中的Voronoi单元估计问题。通过将每个令牌的影响解释为其Voronoi区域的度量,我们的方法实现了保留检索质量的同时减少索引大小的原理化修剪。通过实验,我们证明这种方法不仅是一种具有竞争力的修剪策略,还是一种有价值的工具,用于改进和解释密集检索系统中令牌层面的行为。
英文摘要:
Late-interaction models such as ColBERT offer competitive performance across various retrieval tasks but require storing a dense embedding for each document token, leading to a substantial index storage overhead. Past works address this by attempting to prune low-importance token embeddings based on statistical and empirical measures, but they often either lack formal grounding or are ineffective. To address these shortcomings, we introduce a framework grounded in hyperspace geometry and cast token pruning as a Voronoi cell estimation problem in the embedding space. By interpreting each token's influence as a measure of its Voronoi region, our approach enables principled pruning that retains retrieval quality while reducing index size. Through our experiments, we demonstrate that this approach serves not only as a competitive pruning strategy but also as a valuable tool for improving and interpreting token-level behavior within dense retrieval systems.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。