修正 LogQ 校正：重新审视大规模检索中的采样 Softmax
Correcting the LogQ Correction: Revisiting Sampled Softmax for Large-Scale Retrieval
AI总结：
针对大规模检索中批内负样本 sampled softmax 的 logQ 校正残余偏差，本文指出正样本在分母中必然存在，并提出含错误分类概率权重的精化校正公式，在公开与专有数据集上持续优于标准方法。
摘要：
双塔神经网络是推荐系统检索阶段的一种流行架构。这些模型通常使用基于物品目录的 softmax 损失进行训练。然而，在网络规模场景下，物品目录往往大到令人难以承受，使完整 softmax 不可行。一种常见解决方案是 sampled softmax（采样 softmax），它使用少量采样负样本近似完整 softmax。一种实用且被广泛采用的方法是使用批内负样本，即从当前 mini-batch 中的物品抽取负样本。但这会引入偏差：在批次中出现更频繁的物品（即热门物品）会受到更重的惩罚。为缓解该问题，一种被称为 logQ correction（LogQ 校正）的流行工业技术会在训练期间通过减去物品出现在批次中的对数概率来调整 logits。该校正通过分析梯度中的偏差并应用重要性采样推导而来，实际上是以批内分布作为提议分布进行了两次重要性采样。虽然这种方法提升了模型质量，但并未完全消除偏差。在本工作中，我们重新审视 logQ 校正的推导，并表明它忽略了一个微妙但重要的细节：分母中的正样本物品并非蒙特卡洛采样得到——它始终以概率 1 存在。我们提出了一种考虑这一点的精化校正公式。值得注意的是，我们的损失引入了一个可解释的样本权重，该权重反映模型的不确定性，即当前参数下的错误分类概率。我们在公开数据集和专有数据集上评估了所提方法，结果表明其相较标准 logQ 校正取得了持续提升。
英文摘要：
Two-tower neural networks are a popular architecture for the retrieval stage in recommender systems. These models are typically trained with a softmax loss over the item catalog. However, in web-scale settings, the item catalog is often prohibitively large, making full softmax infeasible. A common solution is sampled softmax, which approximates the full softmax using a small number of sampled negatives. One practical and widely adopted approach is to use in-batch negatives, where negatives are drawn from items in the current mini-batch. However, this introduces a bias: items that appear more frequently in the batch (i.e., popular items) are penalized more heavily. To mitigate this issue, a popular industry technique known as logQ correction adjusts the logits during training by subtracting the log-probability of an item appearing in the batch. This correction is derived by analyzing the bias in the gradient and applying importance sampling, effectively twice, using the in-batch distribution as a proposal distribution. While this approach improves model quality, it does not fully eliminate the bias. In this work, we revisit the derivation of logQ correction and show that it overlooks a subtle but important detail: the positive item in the denominator is not Monte Carlo-sampled - it is always present with probability 1. We propose a refined correction formula that accounts for this. Notably, our loss introduces an interpretable sample weight that reflects the model's uncertainty - the probability of misclassification under the current parameters. We evaluate our method on both public and proprietary datasets, demonstrating consistent improvements over the standard logQ correction.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
