The development of machine learning algorithms in the cyber security domain has been impeded by the complex, hierarchical, sequential and multimodal nature of the data involved. In this paper we introduce the notion of a streaming tree as a generic data structure encompassing a large portion of real-world cyber security data. Starting from host-based event logs we represent computer processes as streaming trees that evolve in continuous time. Leveraging the properties of the signature kernel, a machine learning tool that recently emerged as a leading technology for learning with complex sequences of data, we develop the SK-Tree algorithm. SK-Tree is a supervised learning method for systematic malware detection on streaming trees that is robust to irregular sampling and high dimensionality of the underlying streams. We demonstrate the effectiveness of SK-Tree to detect malicious events on a portion of the publicly available DARPA OpTC dataset, achieving an AUROC score of 98%.
Malware often uses obfuscation techniques or is modified slightly to evade signature detection from antivirus software and malware analysis tools. Traditionally, to determine if a file is malicious and identify what type of malware a sample is, a cryptographic hash of a file is calculated. A more recent and flexible solution for malware detection is YARA, which enables the creation of rules to identify and classify malware based on a file's binary patterns. In this paper, the author will critically evaluate the effectiveness of YARA rules for signature-based detection and classification of mal…
This paper proposes a technique for automatically learning semantic malware signatures for Android from very few samples of a malware family. The key idea underlying our technique is to look for a maximally suspicious common subgraph (MSCS) that is shared between all known instances of a malware family. An MSCS describes the shared functionality between multiple Android applications in terms of inter-component call relations and their semantic metadata (e.g., data-flow properties). Our approach identifies such maximally suspicious common subgraphs by reducing the problem to maximum satisfiabil…
A commonly shared component of antivirus suites is a local database of malware signatures that is used during the static analysis process. Despite possible encryption, heuristic obfuscation, or attempts to hide this database from malicious end-users (or competitors), a currently avoidable eventuality for offline static analysis is a need to use the contents of the database in local computation to detect malicious files. This work serves as a preliminary exploration of the use of concolic execution as a general-case technique for reverse-engineering malware signature database contents: indeed, …
Unsupervised Anomaly-based Malware Detection using Hardware Features
最近的工作已经显示出使用微架构执行模式来检测恶意软件程序的前景。这些检测器属于一类被称为基于签名的检测器的检测器，因为它们通过将程序的执行模式（签名）与已知恶意软件程序的执行模式进行比较来捕获恶意软件。在这项工作中，我们提出了一类新的检测器-基于异常的硬件恶意软件检测器-不需要签名的恶意软件检测，从而可以捕捉到更广泛的恶意软件，包括潜在的新颖的。我们使用无监督的机器学习来基于性能计数器的数据构建正常程序执行的配置文件，并使用这些配置文件来检测由于恶意软件利用而发生的程序行为的重大偏差。我们表明，在Windows/x86平台上的流行程序，如IE和Adobe PDF Reader的现实世界中的剥削，可以检测到几乎完美的确定性。我们还研究了在面对试图逃避基于异常的检测的复杂对手时实施这种方法的局限性和挑战。所提出的检测器是补充以前提出的基于签名的检测器，可以一起使用，以提高安全性。
Although anti-virus software has significantly evolved over the last decade, classic signature matching based on byte patterns is still a prevalent concept for identifying security threats. Anti-virus signatures are a simple and fast detection mechanism that can complement more sophisticated analysis strategies. However, if signatures are not designed with care, they can turn from a defensive mechanism into an instrument of attack. In this paper, we present a novel method for automatically deriving signatures from anti-virus software and demonstrate how the extracted signatures can be used to …
Both malware and antivirus detection tools advance in their capabilities. Malware aim is to evade the detection while antivirus is to detect the malware. Over time, the detection techniques evolved from simple static signature matching over antiheuristic analysis to machine learning assisted algorithms. This thesis describes several layers of anti-virus evasion deployed by the malware and conducts the analysis of the evasion success rate. The scientific contribution of this research is in the following techniques the malware used -- the new algorithm for identifying the Windows operating syste…
We present BPFroid -- a novel dynamic analysis framework for Android that uses the eBPF technology of the Linux kernel to continuously monitor events of user applications running on a real device. The monitored events are collected from different components of the Android software stack: internal kernel functions, system calls, native library functions, and the Java API framework. As BPFroid hooks these events in the kernel, a malware is unable to trivially bypass monitoring. Moreover, using eBPF doesn't require any change to the Android system or the monitored applications. We also present an…
Static detection technologies based on signature-based approaches that are widely used in Android platform to detect malicious applications. It can accurately detect malware by extracting signatures from test data and then comparing the test data with the signature samples of virus and benign samples. However, this method is generally unable to detect unknown malware applications. This is because, sometimes, the machine code can be converted into assembly code, which can be easily read and understood by humans. Furthuremore, the attacker can then make sense of the assembly instructions and und…
Bar-Ilan University（巴伊兰大学）；Tel Aviv University（特拉维夫大学）；Gonda Brain Research Center at Bar-Ilan University（巴伊兰大学贡达脑研究中心）；Center for Automation Research, University of Maryland at College Park（马里兰大学帕克分校自动化研究中心）
This paper presents a novel deep learning based method for automatic malware signature generation and classification. The method uses a deep belief network (DBN), implemented with a deep stack of denoising autoencoders, generating an invariant compact representation of the malware behavior. While conventional signature and token based methods for malware detection do not detect a majority of new variants for existing malware, the results presented in this paper show that signatures generated by the DBN allow for an accurate classification of new malware variants. Using a dataset containing hun…
The challenge in engaging malware activities involves the correct identification and classification of different malware variants. Various malwares incorporate code obfuscation methods that alters their code signatures effectively countering antimalware detection techniques utilizing static methods and signature database. In this study, we utilized an approach of converting a malware binary into an image and use Random Forest to classify various malware families. The resulting accuracy of 0.9562 exhibits the effectivess of the method in detecting malware
Network intrusion detection systems and antivirus software are essential in detecting malicious network traffic and attacks such as denial-of-service and malwares. Each attack, worm or virus has its own distinctive signature. Signature-based intrusion detection and antivirus systems depend on pattern matching to look for possible attack signatures. Pattern matching is a very complex task, which requires a lot of time, memory and computing resources. Software-based intrusion detection is not fast enough to match high network speeds and the increasing number of attacks. In this paper, we propose…
In this paper, we investigate the importance of a defense system's learning rates to fight against the self-propagating class of malware such as worms and bots. To this end, we introduce a new propagation model based on the interactions between an adversary (and its agents) who wishes to construct a zombie army of a specific size, and a defender taking advantage of standard security tools and technologies such as honeypots (HPs) and intrusion detection and prevention systems (IDPSes) in the network environment. As time goes on, the defender can incrementally learn from the collected/observed a…
Universidad Carlos III de Madrid（马德里卡洛斯三世大学）；Telefónica Digital Identity & Privacy（西班牙电信数字身份与隐私部门）
Multi-scanner Antivirus systems provide insightful information on the nature of a suspect application; however there is often a lack of consensus and consistency between different Anti-Virus engines. In this article, we analyze more than 250 thousand malware signatures generated by 61 different Anti-Virus engines after analyzing 82 thousand different Android malware applications. We identify 41 different malware classes grouped into three major categories, namely Adware, Harmful Threats and Unknown or Generic signatures. We further investigate the relationships between such 41 classes using co…
The importance of employing machine learning for malware detection has become explicit to the security community. Several anti-malware vendors have claimed and advertised the application of machine learning in their products in which the inference phase is performed on servers and high-performance machines, but the feasibility of such approaches on mobile devices with limited computational resources has not yet been assessed by the research community, vendors still being skeptical. In this paper, we aim to show the practicality of devising a learning-based anti-malware on Android mobile device…
Malware has become a widely used means in cyber attacks in recent decades because of various new obfuscation techniques used by malwares. In order to protect the systems, data and information, detection of malware is needed as early as possible. There are various studies on malware detection techniques that have been done but there is no method which can detect the malware completely and make malware detection problematic. Static Malware analysis is very effective for known malwares but it does not work for zero day malware which leads to the need of dynamic malware detection and the behaviour…
Current malware detection and classification approaches generally rely on time consuming and knowledge intensive processes to extract patterns (signatures) and behaviors from malware, which are then used for identification. Moreover, these signatures are often limited to local, contiguous sequences within the data whilst ignoring their context in relation to each other and throughout the malware file as a whole. We present a Deep Learning based malware classification approach that requires no expert domain knowledge and is based on a purely data driven approach for complex pattern and feature …
Malwares are the key means leveraged by threat actors in the cyber space for their attacks. There is a large array of commercial solutions in the market and significant scientific research to tackle the challenge of the detection and defense against malwares. At the same time, attackers also advance their capabilities in creating polymorphic and metamorphic malwares to make it increasingly challenging for existing solutions. To tackle this issue, we propose a methodology to perform malware detection and family attribution. The proposed methodology first performs the extraction of opcodes from …
One of the major and serious threats that the Internet faces today is the vast amounts of data and files which need to be evaluated for potential malicious intent. Malicious software, often referred to as a malware that are designed by attackers are polymorphic and metamorphic in nature which have the capability to change their code as they spread. Moreover, the diversity and volume of their variants severely undermine the effectiveness of traditional defenses which typically use signature based techniques and are unable to detect the previously unknown malicious executables. The variants of m…
With the increasingly rapid development of new malicious computer software by bad faith actors, both commercial and research-oriented antivirus detectors have come to make greater use of machine learning tactics to identify such malware as harmful before end users are exposed to their effects. This, in turn, has spurred the development of tools that allow for known malware to be manipulated such that they can evade being classified as dangerous by these machine learning-based detectors, while retaining their malicious functionality. These manipulations function by applying a set of changes tha…
New computing paradigms, modern feature-rich programming languages and off-the-shelf software libraries enabled the development of new sophisticated malware families. Evidence of this phenomena is the recent growth of fileless malware attacks. Fileless malware or memory resident malware is an example of an Advanced Volatile Threat (AVT). In a fileless malware attack, the malware writes itself directly onto the main memory (RAM) of the compromised device without leaving any trace on the compromised device's file system. For this reason, fileless malware presents a difficult challenge for tradit…
Android, the most popular mobile OS, has around 78% of the mobile market share. Due to its popularity, it attracts many malware attacks. In fact, people have discovered around one million new malware samples per quarter, and it was reported that over 98% of these new malware samples are in fact "derivatives" (or variants) from existing malware families. In this paper, we first show that runtime behaviors of malware's core functionalities are in fact similar within a malware family. Hence, we propose a framework to combine "runtime behavior" with "static structures" to detect malware variants. …
While the rapid adaptation of mobile devices changes our daily life more conveniently, the threat derived from malware is also increased. There are lots of research to detect malware to protect mobile devices, but most of them adopt only signature-based malware detection method that can be easily bypassed by polymorphic and metamorphic malware. To detect malware and its variants, it is essential to adopt behavior-based detection for efficient malware classification. This paper presents a system that classifies malware by using common behavioral characteristics along with malware families. We m…
According to the Symantec and F-Secure threat reports, mobile malware development in 2013 and 2014 has continued to focus almost exclusively ~99% on the Android platform. Malware writers are applying stealthy mutations (obfuscations) to create malware variants, thwarting detection by signature based detectors. In addition, the plethora of more sophisticated detectors making use of static analysis techniques to detect such variants operate only at the bytecode level, meaning that malware embedded in native code goes undetected. A recent study shows that 86% of the most popular Android applicati…
Recently, with the booming development of software industry, more and more malware variants are designed to perform malicious behaviors. The evolution of malware makes it difficult to detect using traditional signature-based methods. Moreover, malware detection has important effect on system security. In this paper, we present SCGDet, which is a novel malware detection method based on system call graph model (SCGM). We first develop a system call pruning method, which can exclude system calls that have little impact on malware detection. Then we propose the SCGM, which can capture the semantic…
With the increase in machine learning (ML) applications in different domains, incentives for deceiving these models have reached more than ever. As data is the core backbone of ML algorithms, attackers shifted their interest toward polluting the training data. Data credibility is at even higher risk with the rise of state-of-art research topics like open design principles, federated learning, and crowd-sourcing. Since the machine learning model depends on different stakeholders for obtaining data, there are no reliable automated mechanisms to verify the veracity of data from each source. Malwa…
The most common malware detection approaches which are based on signature matching and are not sufficient for metamorphic malware detection, since virus kits and metamorphic engines can produce variants with no resemblance to one another. Metamorphism provides an efficient way for eluding malware detection software kits. Code obfuscation methods like dead-code insertion are also widely used in metamorphic malware. In order to address the problem of detecting mutated generations, we propose a method based on Opcode Graph Similarity (OGS). OGS tries to detect metamorphic malware using the simila…
Control Flow Graph (CFG) similarity analysis is an essential technique for a variety of security analysis tasks, including malware detection and malware clustering. Even though various algorithms have been developed, existing CFG similarity analysis methods still suffer from limited efficiency, accuracy, and usability. In this paper, we propose a novel fuzzy hashing scheme called topology-aware hashing (TAH) for effective and efficient CFG similarity analysis. Given the CFGs constructed from program binaries, we extract blended n-gram graphical features of the CFGs, encode the graphical featur…
With the growth of mobile devices and applications, the number of malicious software, or malware, is rapidly increasing in recent years, which calls for the development of advanced and effective malware detection approaches. Traditional methods such as signature-based ones cannot defend users from an increasing number of new types of malware or rapid malware behavior changes. In this paper, we propose a new Android malware detection approach based on deep learning and static analysis. Instead of using Application Programming Interfaces (APIs) only, we further analyze the source code of Android…
Computer malware and biological pathogens often use similar mechanisms of infections. For this reason, it has been suggested to model malware spread using epidemiological models developed to characterize the spread of biological pathogens. However, most work examining the similarities between malware and pathogens using such methods was based on theoretical analysis and simulation. Here we extend the classical Susceptible-Infected-Recovered (SIR) epidemiological model to describe two of the most common infection methods used by malware. We fit the proposed model to malware collected over a per…
Malware remains a serious problem for corporations, government agencies, and individuals, as attackers continue to use it as a tool to effect frequent and costly network intrusions. Machine learning holds the promise of automating the work required to detect newly discovered malware families, and could potentially learn generalizations about malware and benign software that support the detection of entirely new, unknown malware families. Unfortunately, few proposed machine learning based malware detection methods have achieved the low false positive rates required to deliver deployable detecto…
National Institute of Technical Teachers Training and Research（国家技术教师培训与研究所）；BITS（比尔拉科学技术学院）
In today's digital world most of the anti-malware tools are signature based which is ineffective to detect advanced unknown malware viz. metamorphic malware. In this paper, we study the frequency of opcode occurrence to detect unknown malware by using machine learning technique. For the purpose, we have used kaggle Microsoft malware classification challenge dataset. The top 20 features obtained from fisher score, information gain, gain ratio, chi-square and symmetric uncertainty feature selection methods are compared. We also studied multiple classifier available in WEKA GUI based machine lear…
As the popularity of Android smart phones has increased in recent years, so too has the number of malicious applications. Due to the potential for data theft mobile phone users face, the detection of malware on Android devices has become an increasingly important issue in cyber security. Traditional methods like signature-based routines are unable to protect users from the ever-increasing sophistication and rapid behavior changes in new types of Android malware. Therefore, a great deal of effort has been made recently to use machine learning models and methods to characterize and generalize th…
Machine learning is a popular approach to signatureless malware detection because it can generalize to never-before-seen malware families and polymorphic strains. This has resulted in its practical use for either primary detection engines or for supplementary heuristic detection by anti-malware vendors. Recent work in adversarial machine learning has shown that deep learning models are susceptible to gradient-based attacks, whereas non-differentiable models that report a score can be attacked by genetic algorithms that aim to systematically reduce the score. We propose a more general framework…
Malware distribution to the victim network is commonly performed through file attachments in phishing email or from the internet, when the victim interacts with the source of infection. To detect and prevent the malware distribution in the victim machine, the existing end device security applications may leverage techniques such as signature or anomaly-based, machine learning techniques. The well-known file formats Portable Executable (PE) for Windows and Executable and Linkable Format (ELF) for Linux based operating system are used for malware analysis, and the malware detection capabilities …
Internet of Things devices have seen a rapid growth and popularity in recent years with many more ordinary devices gaining network capability and becoming part of the ever growing IoT network. With this exponential growth and the limitation of resources, it is becoming increasingly harder to protect against security threats such as malware due to its evolving faster than the defence mechanisms can handle with. The traditional security systems are not able to detect unknown malware as they use signature-based methods. In this paper, we aim to address this issue by introducing a novel IoT malwar…
Nowadays, with the booming development of Internet and software industry, more and more malware variants are designed to perform various malicious activities. Traditional signature-based detection methods can not detect variants of malware. In addition, most behavior-based methods require a secure and isolated environment to perform malware detection, which is vulnerable to be contaminated. In this paper, similar to natural language processing, we propose a novel and efficient approach to perform static malware analysis, which can automatically learn the opcode sequence patterns of malware. We…
With the rapid proliferation and increased sophistication of malicious software (malware), detection methods no longer rely only on manually generated signatures but have also incorporated more general approaches like machine learning detection. Although powerful for conviction of malicious artifacts, these methods do not produce any further information about the type of threat that has been detected neither allows for identifying relationships between malware samples. In this work, we address the information gap between machine learning and signature-based detection methods by learning a repr…
The growing commoditization of the underground economy has given rise to malware delivery networks, which charge fees for quickly delivering malware or unwanted software to a large number of hosts. To provide this service, a key method is the orchestration of silent delivery campaigns, which involve a group of downloaders that receive remote commands and that deliver their payloads without any user interaction. These campaigns have not been characterized systematically, unlike other aspects of malware delivery networks. Moreover, silent delivery campaigns can evade detection by relying on inco…
