结合AI与AM——通过Transformer网络改进近似匹配
Combining AI and AM - Improving Approximate Matching through Transformer Networks
AI总结:
针对传统近似匹配算法小片段检测能力弱、扩展性不足的问题,提出基于Transformer的深度学习近似匹配方法DLAM,可省去恶意片段提取步骤,提升片段恢复准确率,还能辅助传统算法检测小片段下的文件关联。
摘要:
近似匹配(Approximate Matching,AM)是数字取证领域中用于判定数字工件之间相似性的概念。AM的一个重要应用场景是:当仅能获取原始数据的片段时,可靠且高效地检测黑名单中与案件相关的数据结构。例如,在数字取证调查过程中,若仅留存了某一已索引恶意软件的一个簇,AM算法应能将该片段匹配至对应的黑名单恶意软件。然而,传统的AM函数(如TLSH和ssdeep)在待匹配片段相对于文件整体尺寸较小时,无法基于片段识别出对应文件。传统AM算法的第二个公认问题是,随着查找数据库规模不断增长,算法无法实现良好的扩展性。\n我们提出了一种基于自然语言处理领域Transformer模型的改进匹配算法,将其命名为深度学习近似匹配(Deep Learning Approximate Matching,DLAM)。作为人工智能(AI)领域的技术方案,DLAM在训练阶段学习黑名单特征模式的相关知识,随后能够在尺寸通常大得多的文件中检测到这些模式,即DLAM聚焦于片段检测这一应用场景。\n我们的研究表明,与主流传统方法TLSH和ssdeep相比,DLAM具备三大核心优势。第一,DLAM省去了此前使用AM算法搜索前必须完成的、繁琐的已知恶意片段提取步骤,这使得大规模文件高效分类成为可能,而这一点在待调查数据呈指数级增长的当下尤为重要。第二,根据具体应用场景的不同,DLAM在恢复黑名单文件片段方面的准确率与传统方法相当,甚至显著更高。第三,我们证实,即使在片段尺寸极小的情况下,DLAM也能够在TLSH和ssdeep的输出结果中检测出文件间的关联关系。
英文摘要:
Approximate matching (AM) is a concept in digital forensics to determine the similarity between digital artifacts. An important use case of AM is the reliable and efficient detection of case-relevant data structures on a blacklist, if only fragments of the original are available. For instance, if only a cluster of indexed malware is still present during the digital forensic investigation, the AM algorithm shall be able to assign the fragment to the blacklisted malware. However, traditional AM functions like TLSH and ssdeep fail to detect files based on their fragments if the presented piece is relatively small compared to the overall file size. A second well-known issue with traditional AM algorithms is the lack of scaling due to the ever-increasing lookup databases. We propose an improved matching algorithm based on transformer models from the field of natural language processing. We call our approach Deep Learning Approximate Matching (DLAM). As a concept from artificial intelligence (AI), DLAM gets knowledge of characteristic blacklisted patterns during its training phase. Then DLAM is able to detect the patterns in a typically much larger file, that is DLAM focuses on the use case of fragment detection. We reveal that DLAM has three key advantages compared to the prominent conventional approaches TLSH and ssdeep. First, it makes the tedious extraction of known to be bad parts obsolete, which is necessary until now before any search for them with AM algorithms. This allows efficient classification of files on a much larger scale, which is important due to exponentially increasing data to be investigated. Second, depending on the use case, DLAM achieves a similar or even significantly higher accuracy in recovering fragments of blacklisted files. Third, we show that DLAM enables the detection of file correlations in the output of TLSH and ssdeep even for small fragment sizes.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。