arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2303.06389cs.LGcs.AI
收藏

面向离策略学习的不确定性感知实例重加权

Uncertainty-Aware Instance Reweighting for Off-Policy Learning

Bytedance Research(字节跳动研究院);ChongQing University(重庆大学);University of Virginia(弗吉尼亚大学);The Chinese University of Hong Kong(香港中文大学)

Xiaoying Zhang, Junpu Chen, Hongning Wang, Hong Xie, Yang Liu, John C. S. Lui, Hang Li

AI总结:

针对离策略学习中记录策略未知导致的高偏差与高方差问题,本文提出带收敛保证的不确定性感知逆倾向得分估计器UIPS,并在合成及真实推荐数据集上验证其样本效率优势。

摘要:

离策略学习是指仅利用已记录的反馈数据进行策略优化的过程,已在搜索引擎、推荐系统等多种实际应用中展现出重要性。然而,生成这些记录数据的真实记录策略通常是未知的,以往工作在离策略学习中直接使用其估计值,忽视了该估计器带来的高偏差和高方差,尤其是在记录概率较小且估计不准确的样本上。本文显式建模估计记录策略中的不确定性,提出一种不确定性感知逆倾向得分估计器(UIPS),以改进离策略学习,并提供理论收敛保证。在合成数据集和三个真实世界推荐数据集上的实验结果表明,与大量最新基线相比,所提出的UIPS估计器具有更优的样本效率。

英文摘要:

Off-policy learning, referring to the procedure of policy optimization with access only to logged feedback data, has shown importance in various real-world applications, such as search engines, recommender systems, and etc. While the ground-truth logging policy, which generates the logged data, is usually unknown, previous work simply takes its estimated value in off-policy learning, ignoring both high bias and high variance resulted from such an estimator, especially on samples with small and inaccurately estimated logging probabilities. In this work, we explicitly model the uncertainty in the estimated logging policy and propose a Uncertainty-aware Inverse Propensity Score estimator (UIPS) for improved off-policy learning, with a theoretical convergence guarantee. Experiment results on synthetic and three real-world recommendation datasets demonstrate the advantageous sample efficiency of the proposed UIPS estimator against an extensive list of state-of-the-art baselines.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑