契约式强化学习：用看不见的手拉动摇臂
Contractual Reinforcement Learning: Pulling Arms with Invisible Hands
AI总结：
本文提出契约式强化学习框架，用状态依存支付契约协调委托人与智能体利益，并给出最优规划动态规划及可证明低遗憾在线学习算法。
摘要：
代理问题出现在当今的大规模机器学习任务中，此时学习者无法直接指导内容创建或强制执行数据收集。在本文中，我们提出一个理论框架，通过契约设计来协调在线学习问题中不同利益相关者的经济利益。该问题被称为 contractual reinforcement learning（契约式强化学习），自然产生于经典的 Markov 决策过程模型：其中一个学习型委托人寻求通过一组以“下一状态”的实现为条件的支付规则，为了其共同利益而最优地影响智能体的行动策略。对于规划问题，我们设计了一种高效的动态规划算法，以针对远视型智能体确定最优契约。对于学习问题，我们引入了一种通用的 no-regret 学习算法设计，以化解从契约的稳健设计到探索与利用平衡等一系列挑战，并将复杂性分析归结为高效搜索算法的构造。对于若干自然类别的问题，我们设计了定制搜索算法，可证明地实现 $\tilde{O}(\sqrt{T})$ 的遗憾。我们还针对一般问题给出了一种具有 $\tilde{O}(T^{2/3})$ 遗憾的算法，在温和的技术假设下改进了在线契约设计中的现有分析。
英文摘要：
The agency problem emerges in today's large scale machine learning tasks, where the learners are unable to direct content creation or enforce data collection. In this work, we propose a theoretical framework for aligning economic interests of different stakeholders in the online learning problems through contract design. The problem, termed \emph{contractual reinforcement learning}, naturally arises from the classic model of Markov decision processes, where a learning principal seeks to optimally influence the agent's action policy for their common interests through a set of payment rules contingent on the realization of next state. For the planning problem, we design an efficient dynamic programming algorithm to determine the optimal contracts against the far-sighted agent. For the learning problem, we introduce a generic design of no-regret learning algorithms to untangle the challenges from robust design of contracts to the balance of exploration and exploitation, reducing the complexity analysis to the construction of efficient search algorithms. For several natural classes of problems, we design tailored search algorithms that provably achieve $\tilde{O}(\sqrt{T})$ regret. We also present an algorithm with $\tilde{O}(T^{2/3})$ for the general problem that improves the existing analysis in online contract design with mild technical assumptions.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
