arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2603.19339cs.IRcs.AIcs.CL
收藏

光谱温控用于密集路径检索中的嵌入压缩

Spectral Tempering for Embedding Compression in Dense Passage Retrieval

University of Amsterdam(阿姆斯特丹大学)

Yongkang Li, Panagiotis Eustratiadis, Evangelos Kanoulas

AI总结:

本文提出Spectral Tempering方法,通过分析语料库的光谱特性,自适应调整光谱缩放参数γ,实现高效的嵌入压缩,无需标注数据或验证搜索。

摘要:

维度约简对于部署大规模密集检索系统至关重要,但主流后处理方法面临根本性权衡:主成分分析(PCA)保留主导方差但未充分利用表征能力,而白化强制各向同性但放大检索嵌入重尾特征谱中的噪声。中间光谱缩放方法通过用功率系数γ重新加权维度,但将γ视为固定超参数需要任务特定调优。我们表明最优缩放强度γ并非全局常数:它系统性地随目标维度k变化,并由保留子空间的信噪比(SNR)决定。基于此洞察,我们提出Spectral Tempering(SpecTemp),一种无学习方法,通过局部SNR分析和膝点归一化直接从语料库光谱推导出适应性γ(k),无需标注数据或验证基于搜索。大量实验表明,Spectral Tempering在相对网格搜索γ*(k)实现接近 oracle 性能的同时,保持完全无学习和模型无关。我们的代码在https://github.com/liyongkang123/SpecTemp公开。

英文摘要:

Dimensionality reduction is critical for deploying dense retrieval systems at scale, yet mainstream post-hoc methods face a fundamental trade-off: principal component analysis (PCA) preserves dominant variance but underutilizes representational capacity, while whitening enforces isotropy at the cost of amplifying noise in the heavy-tailed eigenspectrum of retrieval embeddings. Intermediate spectral scaling methods unify these extremes by reweighting dimensions with a power coefficient $γ$, but treat $γ$ as a fixed hyperparameter that requires task-specific tuning. We show that the optimal scaling strength $γ$ is not a global constant: it varies systematically with target dimensionality $k$ and is governed by the signal-to-noise ratio (SNR) of the retained subspace. Based on this insight, we propose Spectral Tempering (\textbf{SpecTemp}), a learning-free method that derives an adaptive $γ(k)$ directly from the corpus eigenspectrum using local SNR analysis and knee-point normalization, requiring no labeled data or validation-based search. Extensive experiments demonstrate that Spectral Tempering consistently achieves near-oracle performance relative to grid-searched $γ^*(k)$ while remaining fully learning-free and model-agnostic. Our code is publicly available at https://github.com/liyongkang123/SpecTemp.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑