arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2402.01359cs.LGcs.CRcs.PF
收藏

TESSERACT:消除跨空间与时间的恶意软件分类中的实验偏差(扩展版)

TESSERACT: Eliminating Experimental Bias in Malware Classification across Space and Time (Extended Version)

University College London(伦敦大学学院);HiddenLayer(隐层公司);The Alan Turing Institute(艾伦·图灵研究所);TU Wien(维也纳工业大学);Royal Holloway, University of London(伦敦大学皇家霍洛威学院);LMU Munich(慕尼黑大学)

Zeliang Kan, Shae McFadden, Daniel Arp, Feargus Pendlebury, Roberto Jordaney, Johannes Kinder, Fabio Pierazzi, Lorenzo Cavallaro

AI总结:

本文揭示恶意软件分类中空间与时间偏差导致结果虚高,提出公平实验约束、AUT指标及训练数据调优算法,并推出开源框架TESSERACT,在Android、Windows PE和PDF数据上验证周期性调优可显著提升并稳定检测性能。

摘要:

机器学习在检测恶意软件中发挥着关键作用。尽管大量研究报告的F1分数高达0.99以上,但该问题并未完全解决。由于操作系统和攻击方法不断演变,恶意软件检测器常常出现性能衰减,导致先前学到的知识不足以对新输入做出准确决策。本文认为,通常报告的结果被夸大了,原因在于检测任务中普遍存在的两类实验偏差:空间偏差,源于数据分布不能代表真实部署环境;时间偏差,源于数据的时间划分不当,导致实验配置不切实际。为解决这些偏差,我们引入了一组公平实验设计的约束条件,并提出了一种新指标AUT,用于衡量分类器在真实环境中的鲁棒性。我们还提出了一种算法,旨在调整训练数据以提升分类器性能。最后,我们提出了TESSERACT,一个用于现实分类器比较的开源框架。我们的评估涵盖传统机器学习和深度学习方法,在一个包含259,230个样本、时间跨度为五年的广泛Android数据集上检验了已发表的研究工作。此外,我们还在Windows PE和PDF领域进行了案例研究。我们的发现确认了先前研究中存在偏差,并揭示通过适当的周期性调优可以实现显著的性能提升。我们探索了缓解策略如何通过采用多种策略来延迟性能衰减,从而支持实现更稳定且更优的长期性能。

英文摘要:

Machine learning (ML) plays a pivotal role in detecting malicious software. Despite the high F1-scores reported in numerous studies reaching upwards of 0.99, the issue is not completely solved. Malware detectors often experience performance decay due to constantly evolving operating systems and attack methods, which can render previously learned knowledge insufficient for accurate decision-making on new inputs. This paper argues that commonly reported results are inflated due to two pervasive sources of experimental bias in the detection task: spatial bias caused by data distributions that are not representative of a real-world deployment; and temporal bias caused by incorrect time splits of data, leading to unrealistic configurations. To address these biases, we introduce a set of constraints for fair experiment design, and propose a new metric, AUT, for classifier robustness in real-world settings. We additionally propose an algorithm designed to tune training data to enhance classifier performance. Finally, we present TESSERACT, an open-source framework for realistic classifier comparison. Our evaluation encompasses both traditional ML and deep learning methods, examining published works on an extensive Android dataset with 259,230 samples over a five-year span. Additionally, we conduct case studies in the Windows PE and PDF domains. Our findings identify the existence of biases in previous studies and reveal that significant performance enhancements are possible through appropriate, periodic tuning. We explore how mitigation strategies may support in achieving a more stable and better performance over time by employing multiple strategies to delay performance decay.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑