从稳定性测试中精确定位网络中的异常事件-- N-Grams与深度学习
Pinpointing Anomaly Events in Logs from Stability Testing -- N-Grams vs. Deep-Learning
摘要：
由于稳定性测试执行日志可能很长，软件工程师需要帮助定位异常事件。我们开发并评估了两个模型，用于对单个日志事件进行评分，即N-Gram模型和具有LSTM（长短期记忆）的深度学习模型。两者都只在正常的日志序列上训练。我们评估的模型与长日志序列的Android稳定性测试在我们公司的情况下，和短日志序列的HDFS（Hadoop分布式文件系统）的公共数据集。我们评估下一个事件的预测精度和计算效率。LSTM模型在稳定性测试日志中更准确（0.848 vs 0.865），而在HDFS日志中，N-Gram稍微更准确（0.904 vs 0.900）。与Deep模型相比，N-Gram模型具有更高的计算效率（4到13秒vs 16分钟到近4小时），使其成为我们案例公司的首选。对单个日志事件的错误性进行评分似乎是对失败测试用例的根本原因分析的一个很好的帮助，我们的案例公司计划将其添加到其在线服务中。尽管最近在软件系统异常检测中使用深度学习的浪潮激增，但我们发现这样做的好处有限。然而，未来的工作应该考虑我们的发现是否适用于不同的LSTM模型超参数，其他数据集，以及其他深度学习方法，这些方法比基于LSTM的模型具有更好的准确性和计算效率。
英文摘要：
As stability testing execution logs can be very long, software engineers need help in locating anomalous events. We develop and evaluate two models for scoring individual log-events for anomalousness, namely an N-Gram model and a Deep Learning model with LSTM (Long short-term memory). Both are trained on normal log sequences only. We evaluate the models with long log sequences of Android stability testing in our company case and with short log sequences from HDFS (Hadoop Distributed File System) public dataset. We evaluate next event prediction accuracy and computational efficiency. The LSTM model is more accurate in stability testing logs (0.848 vs 0.865), whereas in HDFS logs the N-Gram is slightly more accurate (0.904 vs 0.900). The N-Gram model has far superior computational efficiency compared to the Deep model (4 to 13 seconds vs 16 minutes to nearly 4 hours), making it the preferred choice for our case company. Scoring individual log events for anomalousness seems like a good aid for root cause analysis of failing test cases, and our case company plans to add it to its online services. Despite the recent surge in using deep learning in software system anomaly detection, we found limited benefits in doing so. However, future work should consider whether our finding holds with different LSTM-model hyper-parameters, other datasets, and with other deep-learning approaches that promise better accuracy and computational efficiency than LSTM based models.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
