基于专家混合的残差量化：用于高效输入依赖向量压缩
RQ-MoE: Residual Quantization via Mixture of Experts for Efficient Input-Dependent Vector Compression
AI总结：
本文提出RQ-MoE框架，结合双层专家混合与双流量化，实现输入依赖的代码本自适应，提升向量量化效率，并在解码速度上优于现有方法。
摘要：
向量量化是压缩高维嵌入的基础工具，但现有多代码本方法依赖静态代码本，限制了异质数据几何下的表达能力。尽管近期动态量化器如QINCo能适应个体输入并提升表达性，但其严格的顺序依赖性导致解码瓶颈。本文提出基于专家混合的残差量化（RQ-MoE），通过双层专家混合与双流量化实现输入依赖的代码本自适应，以实现高效的向量量化。RQ-MoE支持动态代码本构建，并将指令与量化解耦，促进并行解码。理论上，我们证明标准残差量化和QINCo可以作为RQ-MoE的约束特例，并推导了RQ-MoE中专家维度设置的指导方针。大量实验表明，RQ-MoE在重建和检索任务中达到最先进的性能或与现有方法相当，同时比现有向量量化方法快6至14倍。实现代码可在https://github.com/KDEGroup/RQ-MoE上获取。
英文摘要：
Vector quantization is a fundamental tool for compressing high-dimensional embeddings, yet existing multi-codebook methods rely on static codebooks that limit expressiveness under heterogeneous data geometry. While recent dynamic quantizers like QINCo adapt codebooks to individual inputs and improve expressiveness, their strict sequential dependencies create decoding bottlenecks. We propose Residual Quantization via Mixture of Experts (RQ-MoE), a framework combining a two-level MoE with dual-stream quantization to enable input-dependent codebook adaptation for efficient vector quantization. RQ-MoE enables dynamic codebook construction and decouples instruction from quantization, facilitating parallel decoding. Theoretically, we show that standard Residual Quantization and QINCo can be recovered as constrained special cases of RQ-MoE, and derive a guideline for setting expert dimensionality in RQ-MoE. Extensive experiments show that RQ-MoE achieves state-of-the-art or on-par performance in reconstruction and retrieval, while providing 6x-14x faster decoding than prior vector quantization methods. The implementation is available at https://github.com/KDEGroup/RQ-MoE.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
