Anomalies or failures in large computer systems, such as the cloud, have an impact on a large number of users that communicate, compute, and store information. Therefore, timely and accurate anomaly detection is necessary for reliability, security, safe operation, and mitigation of losses in these increasingly important systems. Recently, the evolution of the software industry opens up several problems that need to be tackled including (1) addressing the software evolution due software upgrades, and (2) solving the cold-start problem, where data from the system of interest is not available. In this paper, we propose a framework for anomaly detection in log data, as a major troubleshooting source of system information. To that end, we utilize pre-trained general-purpose language models to preserve the semantics of log messages and map them into log vector embeddings. The key idea is that these representations for the logs are robust and less invariant to changes in the logs, and therefore, result in a better generalization of the anomaly detection models. We perform several experiments on a cloud dataset evaluating different language models for obtaining numerical log representations such as BERT, GPT-2, and XL. The robustness is evaluated by gradually altering log messages, to simulate a change in semantics. Our results show that the proposed approach achieves high performance and robustness, which opens up possibilities for future research in this direction.
Topological Data Analysis for Anomaly Detection in Host-Based Logs
The Alan Turing Institute(艾伦·图灵研究所);The British Library(大英图书馆)
拓扑数据分析(TDA)使从业人员能够分析网络安全数据的全局结构。我们使用TDA在基于主机的日志中进行异常检测,这些日志是通过开源Logging Made Easy(LME)项目收集的。我们提出了一种方法,直接从Windows日志中建立单纯复合体的过滤,使其内在结构的分析,使用拓扑工具。我们比较持久的同源性和频谱图和超图拉普拉斯算子作为特征向量对标准日志嵌入计数事件的功效,并发现计算机日志的拓扑和频谱嵌入包含用于对异常日志进行分类的判别信息,该信息是标准嵌入的补充。最后,我们讨论了我们的方法被用作异常检测的一个可解释的框架的一部分的潜力。
LogAction: Consistent Cross-system Anomaly Detection through Logs via Active Domain Adaptation
School of Software(软件学院);Institute for Artificial Intelligence, Peking University, Beijing, China(人工智能研究院,北京大学,北京,中国);National Engineering Research Center for Software Engineering, Peking University, Beijing, China(软件工程国家工程研究中心,北京大学,北京,中国);National Key Laboratory of Data Space Technology(数据空间技术与系统国家重点实验室)
TMUAD: Enhancing Logical Capabilities in Unified Anomaly Detection Models with a Text Memory Bank
Shenyang Institute of Automation, Chinese Academy of Sciences(中国科学院沈阳自动化研究所);Liaoning Liaohe Laboratory(辽宁省辽河实验室);Key Laboratory on Intelligent Detection and Equipment Technology(智能检测与装备技术重点实验室);University of Chinese Academy of Sciences(中国科学院大学);College of Automation Science and Engineering(自动化科学与工程学院);State Key Laboratory of Robotics and Intelligent Systems(机器人与智能系统国家重点实验室);South China University of Technology(华南理工大学)
ContraLog: Log File Anomaly Detection with Contrastive Learning and Masked Language Modeling
Munich Center for Machine Learning(慕尼黑机器学习中心);LMU München(慕尼黑大学);German Research Center for Environmental Health, Helmholtz Zentrum München(德国环境健康研究中心,亥姆霍兹慕尼黑中心)
TransLog: A Unified Transformer-based Framework for Log Anomaly Detection
State Key Lab of Software Development Environment, Beihang University(北京航空航天大学软件开发环境国家重点实验室);Cloudwise Research(云智慧研究院);University of Southern California(南加州大学);Bio-robot and human-computer interaction laboratory, Waseda University(早稻田大学生物机器人与人机交互实验室)
RationAnomaly: Log Anomaly Detection with Rationality via Chain-of-Thought and Reinforcement Learning
School of Software Engineering, University of Science and Technology of China(软件工程学院,中国科学技术大学);Suzhou Institute for Advanced Research, University of Science and Technology of China(苏州市先进研究 institute,中国科学技术大学);Huawei, Beijing, China(华为,中国);Huawei Canada, Montreal, Canada(华为加拿大分公司,加拿大);Nankai University, Tianjin, China(南开大学,中国)
针对现有日志异常检测方法缺乏可解释性或存在事实不可靠的问题,本文提出 RationAnomaly 框架,结合思维链监督微调与多方面奖励的强化学习,在提升 F1 分数的同时提供透明可解释的逐步推理输出。
Quantum Information Fusion and Correction under the Transferable Belief Model
School of Electronics and Information, Northwestern Polytechnical University(电子信息学院,西北工业大学);College of Future Information Technology, Fudan University(未来信息技术学院,复旦大学);College of Information Engineering, Northwest A\&F University(信息工程学院,西北农林科技大学);Institute of Fundamental and Frontier Science, University of Electronic Science and Technology of China(基础与前沿科学研究院,电子科技大学);Department of Image and Information Processing, IMT-Atlantique(图像与信息处理系,IMT-Atlantic)