We address the problem of finding an optimal policy in a Markov decision process under a restricted policy class defined by the convex hull of a set of base policies. This problem is of great interest in applications in which a number of reasonably good (or safe) policies are already known and we are only interested in optimizing in their convex hull. We show that this problem is NP-hard to solve exactly as well as to approximate to arbitrary accuracy. However, under a condition that is akin to the occupancy measures of the base policies having large overlap, we show that there exists an efficient algorithm that finds a policy that is almost as good as the best convex combination of the base policies. The running time of the proposed algorithm is linear in the number of states and polynomial in the number of base policies. In practice, we demonstrate an efficient implementation for large state problems. Compared to traditional policy gradient methods, the proposed approach has the advantage that, apart from the computation of occupancy measures of some base policies, the iterative method need not interact with the environment during the optimization process. This is especially important in complex systems where estimating the value of a policy can be a time consuming process.
Learning Markov Decision Processes under Fully Bandit Feedback
Department of Industrial and Operations Engineering, University of Michigan（工业与运作工程系，密歇根大学）；Computer Science Department, New York University（计算机科学系，纽约大学）
Streaming Methods for Restricted Strongly Convex Functions with Applications to Prototype Selection
Amazon Development Center（亚马逊开发中心）；IBM Research（IBM研究院）
在本文中，我们表明，如果优化功能是限制强凸（RSC）和限制光滑（RSM）-一个丰富的子类弱次模函数-然后流算法与常数因子近似保证是可能的。更一般地说，我们的结果适用于任何单调弱次模函数的子模比有界从上。这个（正）结果为弱次模函数提供了一个常数因子流保证的充分条件，考虑到最近的负结果，它可能是特别令人感兴趣的（Elenberg et al.，2017）对于弱次模函数的一般类。我们应用我们的流算法创建大型复杂数据集的紧凑的概要，通过选择$m$代表元素，通过优化合适的RSC和RSM目标函数。上述结果甚至在附加约束条件下也成立，例如学习非负权重，用于可解释性，用于指示其重要性的每个所选元素。我们在两个真实的数据集上对我们的算法进行了经验评估：MNIST-手写数字数据集和Letters-包含以不同字体和样式书写的字母表的UCI数据集。我们观察到，我们的算法是数量级的速度比最先进的流算法弱次模函数，我们的主要算法仍然提供同样好的解决方案在实践中。
