arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2307.09388cs.LG
收藏

非平稳环境下特征获取存在成本的在线学习

Online Learning with Costly Features in Non-stationary Environments

Tübingen University(图宾根大学);Fraunhofer Heinrich Herz Institute(弗劳恩霍夫海因里希·赫兹研究所)

Saeed Ghoorchian, Evgenii Kortukov, Setareh Maghsudi

AI总结:

针对非平稳环境下特征获取需成本的序贯决策问题,拓展上下文老虎机设置,提出一种保证次线性遗憾值的算法,在真实场景中实现更优的长期平均收益。

摘要:

最大化长期奖励是序贯决策问题的核心目标。现有大多数方法均假设辅助信息可免费获取,允许学习智能体在决策前观测所有特征的状态。但在实际问题中,收集有效信息往往需要付出成本。这意味着,除了各臂的奖励之外,学习特征状态的观测结果对于优化决策策略至关重要。在奖励和成本分布会随时间发生突变的非平稳环境中,该问题会变得更加棘手。为解决上述双重学习问题,我们拓展了上下文老虎机(contextual bandit)设置,允许智能体观测特征状态的子集。研究目标是最大化长期平均收益,即累计奖励与已支付成本的平均差值。因此,智能体需要在最小化信息获取成本与利用所得信息优化决策过程之间进行权衡。为此,我们提出了一种算法,该算法可保证 regret(遗憾值)随时间呈次线性增长。数值实验结果表明,我们提出的策略在真实场景中具有更优的性能。

英文摘要:

Maximizing long-term rewards is the primary goal in sequential decision-making problems. The majority of existing methods assume that side information is freely available, enabling the learning agent to observe all features' states before making a decision. In real-world problems, however, collecting beneficial information is often costly. That implies that, besides individual arms' reward, learning the observations of the features' states is essential to improve the decision-making strategy. The problem is aggravated in a non-stationary environment where reward and cost distributions undergo abrupt changes over time. To address the aforementioned dual learning problem, we extend the contextual bandit setting and allow the agent to observe subsets of features' states. The objective is to maximize the long-term average gain, which is the difference between the accumulated rewards and the paid costs on average. Therefore, the agent faces a trade-off between minimizing the cost of information acquisition and possibly improving the decision-making process using the obtained information. To this end, we develop an algorithm that guarantees a sublinear regret in time. Numerical results demonstrate the superiority of our proposed policy in a real-world scenario.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑