arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2402.04567cs.LGcs.AI
收藏

OIL-AD:面向序贯决策序列的异常检测框架

OIL-AD: An Anomaly Detection Framework for Sequential Decision Sequences

The University of Melbourne(墨尔本大学)

Chen Wang, Sarah Erfani, Tansu Alpcan, Christopher Leckie

AI总结:

提出无监督的OIL-AD框架,利用离线模仿学习从正常轨迹中学习Q函数和状态值函数,提取动作最优性和序贯关联性特征,实现决策序列的在线异常检测,F1分数最高提升34.8%。

摘要:

决策序列中的异常检测是一个具有挑战性的问题,原因在于正常性表示学习的复杂性以及任务的序贯特性。大多数基于强化学习(RL)的现有方法由于不切实际的假设(例如能够访问环境动态、奖励信号以及与环境的在线交互)而难以在现实世界中实施。为解决这些局限,我们提出了一种名为基于离线模仿学习的异常检测(OIL-AD)的无监督方法,该方法利用两个提取的行为特征(动作最优性和序贯关联性)来检测决策序列中的异常。我们的离线学习模型是对行为克隆与Transformer策略网络的一种改编,其中我们修改了训练过程,以从正常轨迹中学习Q函数和状态值函数。我们提出,Q函数和状态值函数能够提供关于智能体行为数据的充分信息,从中我们推导出两个用于异常检测的特征。我们方法的直觉在于,从Q函数推导出的动作最优性特征能够在每个局部状态区分最优动作与其他动作,而从状态值函数推导出的序贯关联性特征则有可能保持决策(状态-动作对)之间的时间相关性。我们的实验表明,OIL-AD能够实现出色的在线异常检测性能,在F1分数上相较于可比基线最高提升34.8%。

英文摘要:

Anomaly detection in decision-making sequences is a challenging problem due to the complexity of normality representation learning and the sequential nature of the task. Most existing methods based on Reinforcement Learning (RL) are difficult to implement in the real world due to unrealistic assumptions, such as having access to environment dynamics, reward signals, and online interactions with the environment. To address these limitations, we propose an unsupervised method named Offline Imitation Learning based Anomaly Detection (OIL-AD), which detects anomalies in decision-making sequences using two extracted behaviour features: action optimality and sequential association. Our offline learning model is an adaptation of behavioural cloning with a transformer policy network, where we modify the training process to learn a Q function and a state value function from normal trajectories. We propose that the Q function and the state value function can provide sufficient information about agents' behavioural data, from which we derive two features for anomaly detection. The intuition behind our method is that the action optimality feature derived from the Q function can differentiate the optimal action from others at each local state, and the sequential association feature derived from the state value function has the potential to maintain the temporal correlations between decisions (state-action pairs). Our experiments show that OIL-AD can achieve outstanding online anomaly detection performance with up to 34.8% improvement in F1 score over comparable baselines.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑