arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2209.08754cs.LG
收藏

对学习排序中特征提取的理解

Toward Understanding Privileged Features Distillation in Learning-to-Rank

UT Austin(得克萨斯大学奥斯汀分校);Amazon(亚马逊)

Shuo Yang, Sujay Sanghavi, Holakou Rahmanian, Jan Bakus, S. V. N. Vishwanathan

摘要:

在学习排序问题中,特权特征是在模型训练期间可用,但在测试时不可用的特征。这样的特征自然出现在商品化推荐系统中;例如,作为特征的“用户点击了该物品”在离线数据中预测“用户购买了该物品”,但是在在线服务期间显然不可用。特权特性的另一个来源是那些在线计算成本太高但可以离线添加的特性。特征提取(PFD)是指一种自然的想法:训练一个使用所有特征(包括特权特征)的“教师”模型,然后用它来训练一个不使用特权特征的“学生”模型。在本文中,我们首先在三个公共排名数据集和一个来自亚马逊日志的工业规模排名问题上实证研究PFD。我们表明,PFD优于几个基线(无蒸馏,预训练微调,自蒸馏和广义蒸馏)在所有这些数据集。接下来,我们通过经验消融研究和线性模型的理论分析来分析PFD为什么以及何时表现良好。这两项研究都揭示了一个有趣的非单调行为:随着特权特征的预测能力增加,最终学生模型的性能最初会增加,但随后会下降。我们发现,后来的性能下降的原因是,一个非常具有预测性的特权教师产生的预测与高方差,这导致高方差的学生估计和较差的测试性能。

英文摘要:

In learning-to-rank problems, a privileged feature is one that is available during model training, but not available at test time. Such features naturally arise in merchandised recommendation systems; for instance, "user clicked this item" as a feature is predictive of "user purchased this item" in the offline data, but is clearly not available during online serving. Another source of privileged features is those that are too expensive to compute online but feasible to be added offline. Privileged features distillation (PFD) refers to a natural idea: train a "teacher" model using all features (including privileged ones) and then use it to train a "student" model that does not use the privileged features. In this paper, we first study PFD empirically on three public ranking datasets and an industrial-scale ranking problem derived from Amazon's logs. We show that PFD outperforms several baselines (no-distillation, pretraining-finetuning, self-distillation, and generalized distillation) on all these datasets. Next, we analyze why and when PFD performs well via both empirical ablation studies and theoretical analysis for linear models. Both investigations uncover an interesting non-monotone behavior: as the predictive power of a privileged feature increases, the performance of the resulting student model initially increases but then decreases. We show the reason for the later decreasing performance is that a very predictive privileged teacher produces predictions with high variance, which lead to high variance student estimates and inferior testing performance.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑