This paper introduces von Mises-Fisher exploration (vMF-exp), a scalable method for exploring large action sets in reinforcement learning problems where hyperspherical embedding vectors represent these actions. vMF-exp involves initially sampling a state embedding representation using a von Mises-Fisher distribution, then exploring this representation's nearest neighbors, which scales to virtually unlimited numbers of candidate actions. We show that, under theoretical assumptions, vMF-exp asymptotically maintains the same probability of exploring each action as Boltzmann Exploration (B-exp), a popular alternative that, nonetheless, suffers from scalability issues as it requires computing softmax values for each action. Consequently, vMF-exp serves as a scalable alternative to B-exp for exploring large action sets with hyperspherical embeddings. Experiments on simulated data, real-world public data, and the successful large-scale deployment of vMF-exp on the recommender system of a global music streaming service empirically validate the key properties of the proposed method.
Variable Selection with Rigorous Uncertainty Quantification using Deep Bayesian Neural Networks: Posterior Concentration and Bernstein-von Mises Phenomenon
On PyTorch Implementation of Density Estimators for von Mises-Fisher and Its Mixture
von Mises-Fisher(vMF)是一个著名的方向随机变量密度模型。最近兴起的用于高维结构化数据(如图像或文本)的深度嵌入方法,旨在提取突出的方向信息,可以使vMF模型更加流行。在本文中,我们将回顾vMF模型及其混合,提供如何在Python/PyTorch中训练模型的详细方法,重点是最大似然估计。特别是,vMF的实现通常会遇到密度归一化器中贝塞尔函数求值的臭名昭著的数值问题,特别是当维数很高时,我们使用支持任意精度的MPMath库来解决这个问题。对于混合学习,我们提供了基于小批量的大规模SGD学习,以及EM算法,这是一个完整的批量估计器。对于每种估计器/方法,我们在一些合成数据上测试我们的实现,同时我们还在更现实的图像聚类场景中演示了用例。我们的代码可在https://github.com/minyoungkim21/vmf-lib上公开获取。