多探针零碰撞哈希（MPZCH）：在大规模推荐系统中缓解嵌入碰撞并提升模型新鲜度
Multi-Probe Zero Collision Hash (MPZCH): Mitigating Embedding Collisions and Enhancing Model Freshness in Large-Scale Recommenders
AI总结：
本文提出MPZCH，一种基于线性探测的新型索引机制，通过合理表大小有效消除嵌入碰撞，保持生产效率，利用辅助张量和CUDA内核实现可配置探测和主动驱逐策略，防止哈希方法中的陈旧嵌入继承，提升物品嵌入新鲜度和质量。
摘要：
嵌入表是大规模推荐系统的关键组件，有助于高效地将高基数的类别特征映射为密集向量表示。然而，随着唯一ID数量的增加，传统基于哈希的索引方法会因碰撞而降级模型性能和个性化质量。我们提出了多探针零碰撞哈希（MPZCH），一种基于线性探测的新索引机制，能够有效缓解嵌入碰撞。通过合理的表大小，它通常能够完全消除这些碰撞，同时保持大规模生产效率。MPZCH利用辅助张量和高性能CUDA内核实现可配置探测和主动驱逐策略。通过淘汰过时的ID并重置重新分配的槽位，MPZCH防止了哈希方法中典型的陈旧嵌入继承，确保新特征能够从头开始有效学习。尽管有缓解碰撞的开销，该系统保持了训练QPS和推理延迟与现有方法相当。严格在线实验表明，MPZCH实现了用户嵌入的零碰撞，并显著提高了物品嵌入的新鲜度和质量。该解决方案已发布在开源TorchRec库中，供更广泛的社区使用。
英文摘要：
Embedding tables are critical components of large-scale recommendation systems, facilitating the efficient mapping of high-cardinality categorical features into dense vector representations. However, as the volume of unique IDs expands, traditional hash-based indexing methods suffer from collisions that degrade model performance and personalization quality. We present Multi-Probe Zero Collision Hash (MPZCH), a novel indexing mechanism based on linear probing that effectively mitigates embedding collisions. With reasonable table sizing, it often eliminates these collisions entirely while maintaining production-scale efficiency. MPZCH utilizes auxiliary tensors and high-performance CUDA kernels to implement configurable probing and active eviction policies. By retiring obsolete IDs and resetting reassigned slots, MPZCH prevents the stale embedding inheritance typical of hash-based methods, ensuring new features learn effectively from scratch. Despite its collision-mitigation overhead, the system maintains training QPS and inference latency comparable to existing methods. Rigorous online experiments demonstrate that MPZCH achieves zero collisions for user embeddings and significantly improves item embedding freshness and quality. The solution has been released within the open-source TorchRec library for the broader community.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
