通过具有自动模型选择的分层非负矩阵分解解决极端类别不平衡下的恶意软件家族半监督分类
Semi-supervised Classification of Malware Families Under Extreme Class Imbalance via Hierarchical Non-Negative Matrix Factorization with Automatic Model Selection
AI总结：
本文提出一种名为 HNMFk Classifier 的分层半监督算法，基于带自动模型选择的非负矩阵分解，通过支持弃权（不执行）预测，在极端类别不平衡和标注数据稀缺的情况下有效分类恶意软件家族并识别新型家族，在 EMBER-2018 数据集上取得 0.80 的 F1 分数。
摘要：
识别恶意软件样本所属的家族对于理解恶意软件行为和制定缓解策略至关重要。然而，由于缺乏现实的评估因素，先前工作提出的解决方案通常不可行。这些因素包括类别不平衡下的学习、识别新型恶意软件的能力以及生产级标注数据的成本。在实践中，部署的模型面临突出的、罕见的以及新型的恶意软件家族。同时，获取大量最新的标注恶意软件来训练模型可能非常昂贵。在本文中，我们解决了这些问题，并提出了一种新颖的分层半监督算法，我们称之为 HNMFk Classifier，可用于恶意软件家族标注过程的早期阶段。我们的方法基于带有自动模型选择的非负矩阵分解，即对聚类数量进行估计。通过 HNMFk Classifier，我们利用恶意软件数据的分层结构以及半监督设置，使我们能够在极端类别不平衡的条件下对恶意软件家族进行分类。我们的解决方案可以执行弃权（不执行）预测或拒绝选项，这在识别新型恶意软件家族方面产生了有希望的结果，并有助于在使用少量标注数据时维持模型性能。我们通过静态分析，使用来自 EMBER-2018 语料库的近 388,000 个样本，对近 2,900 个罕见和突出的恶意软件家族进行了批量分类。在我们的实验中，我们以 0.80 的 F1 分数超越了监督和半监督基线模型。
英文摘要：
Identification of the family to which a malware specimen belongs is essential in understanding the behavior of the malware and developing mitigation strategies. Solutions proposed by prior work, however, are often not practicable due to the lack of realistic evaluation factors. These factors include learning under class imbalance, the ability to identify new malware, and the cost of production-quality labeled data. In practice, deployed models face prominent, rare, and new malware families. At the same time, obtaining a large quantity of up-to-date labeled malware for training a model can be expensive. In this paper, we address these problems and propose a novel hierarchical semi-supervised algorithm, which we call the HNMFk Classifier, that can be used in the early stages of the malware family labeling process. Our method is based on non-negative matrix factorization with automatic model selection, that is, with an estimation of the number of clusters. With HNMFk Classifier, we exploit the hierarchical structure of the malware data together with a semi-supervised setup, which enables us to classify malware families under conditions of extreme class imbalance. Our solution can perform abstaining predictions, or rejection option, which yields promising results in the identification of novel malware families and helps with maintaining the performance of the model when a low quantity of labeled data is used. We perform bulk classification of nearly 2,900 both rare and prominent malware families, through static analysis, using nearly 388,000 samples from the EMBER-2018 corpus. In our experiments, we surpass both supervised and semi-supervised baseline models with an F1 score of 0.80.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
