Anomaly detection is a fundamental problem in data mining field with many real-world applications. A vast majority of existing anomaly detection methods predominately focused on data collected from a single source. In real-world applications, instances often have multiple types of features, such as images (ID photos, finger prints) and texts (bank transaction histories, user online social media posts), resulting in the so-called multi-modal data. In this paper, we focus on identifying anomalies whose patterns are disparate across different modalities, i.e., cross-modal anomalies. Some of the data instances within a multi-modal context are often not anomalous when they are viewed separately in each individual modality, but contains inconsistent patterns when multiple sources are jointly considered. The existence of multi-modal data in many real-world scenarios brings both opportunities and challenges to the canonical task of anomaly detection. On the one hand, in multi-modal data, information of different modalities may complement each other in improving the detection performance. On the other hand, complicated distributions across different modalities call for a principled framework to characterize their inherent and complex correlations, which is often difficult to capture with conventional linear models. To this end, we propose a novel deep structured anomaly detection framework to identify the cross-modal anomalies embedded in the data. Experiments on real-world datasets demonstrate the effectiveness of the proposed framework comparing with the state-of-the-art.
Binghamton University（宾汉姆顿大学）；IBM T. J. Watson Research Center（IBM托马斯·J·沃森研究中心）；Tsinghua University（清华大学）
In this paper, we attack the anomaly detection problem by directly modeling the data distribution with deep architectures. We propose deep structured energy based models (DSEBMs), where the energy function is the output of a deterministic deep neural network with structure. We develop novel model architectures to integrate EBMs with different types of data such as static data, sequential data, and spatial data, and apply appropriate model architectures to adapt to the data structure. Our training algorithm is built upon the recent development of score matching \cite{sm}, which connects an EBM …
Harvard University（哈佛大学）；U.S. Army Research Laboratory（美国陆军研究实验室）；Booz Allen Hamilton（博思艾伦汉密尔顿）；Duke University（杜克大学）
A framework is proposed to detect anomalies in multi-modal data. A deep neural network-based object detector is employed to extract counts of objects and sub-events from the data. A cyclostationary model is proposed to model regular patterns of behavior in the count sequences. The anomaly detection problem is formulated as a problem of detecting deviations from learned cyclostationary behavior. Sequential algorithms are proposed to detect anomalies using the proposed model. The proposed algorithms are shown to be asymptotically efficient in a well-defined sense. The developed algorithms are ap…
School of Computer Science and Engineering, Beihang University（北京航空航天大学计算机科学与工程学院）；School of Software, Beihang University（北京航空航天大学软件学院）；Shandong Inspur Intelligent Production Technology Co., Ltd（山东浪潮智能生产技术有限公司）
RobustA: Robust Anomaly Detection in Multimodal Data
Mohamed Bin Zayed University of Artificial Intelligence（穆罕默德·本·扎耶德人工智能大学）；IMT School for Advanced Studies（IMT高级研究学院）；Johannes Kepler University Linz（林茨约瑟夫·冯·克莱门茨大学）；Human-centered AI Group, AI Lab, Linz Institute of Technology（以人为中心的人工智能小组、人工智能实验室、林茨技术研究所）
Harbin University of Science and Technology（哈尔滨理工大学）；Minjiang University（闽江学院）
Anomaly detection on attributed networks aims at finding nodes whose patterns deviate significantly from the majority of reference nodes, which is pervasive in many applications such as network intrusion detection and social spammer detection. However, most existing methods neglect the complex cross-modality interactions between network structure and node attribute. In this paper, we propose a deep joint representation learning framework for anomaly detection through a dual autoencoder (AnomalyDAE), which captures the complex interactions between network structure and node attribute for high-q…
University of Illinois at Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）
To achieve high-levels of autonomy, modern robots require the ability to detect and recover from anomalies and failures with minimal human supervision. Multi-modal sensor signals could provide more information for such anomaly detection tasks; however, the fusion of high-dimensional and heterogeneous sensor modalities remains a challenging problem. We propose a deep learning neural network: supervised variational autoencoder (SVAE), for failure identification in unstructured and uncertain environments. Our model leverages the representational power of VAE to extract robust features from high-d…
Moon: A Modality Conversion-based Efficient Multivariate Time Series Anomaly Detection
College of Computer Science, Zhejiang University（浙江大学计算机科学学院）；Department of Computer and Information Science, University of Macau（澳门大学计算机与信息科学系）；Department of Computer Science, Aalborg University（奥胡斯大学计算机科学系）
Spatiotemporal traffic data (e.g., link speed/flow) collected from sensor networks can be organized as multivariate time series with additional spatial attributes. A crucial task in analyzing such data is to identify and detect anomalous observations and events from the data with complex spatial and temporal dependencies. Robust Principal Component Analysis (RPCA) is a widely used tool for anomaly detection. However, the traditional RPCA purely relies on the global low-rank assumption while ignoring the local temporal correlations. In light of this, this study proposes a Hankel-structured tens…
Robust Anomaly Detection through Multi-Modal Autoencoder Fusion for Small Vehicle Damage Detection
Faculty of Mathematics and Computer Science, University of Bremen（数学与计算机科学学院，不莱梅大学）；Robotics Innovation Center, Deutsches Forschungszentrum für Künstliche Intelligenz（机器人创新中心，德国人工智能研究中心）；Engineering Software Communication, Robert Bosch GmbH（工程软件通信，罗伯特·博世有限公司）
UTOPYA: A Multimodal Deep Learning Framework for Physics-Informed Anomaly Detection and Time-Series Prediction
Department of Chemical Engineering, Norwegian University of Science and Technology (NTNU)（化学工程系，挪威科学与技术大学）；Department of Computing, Imperial College London（计算系，帝国理工学院伦敦分校）
VACE: Learning Geometrically Structured Representations for Time Series Anomaly Detection
Andalusian Research Institute in Data Science and Computational Intelligence (DaSCI)（安达卢西亚数据科学与计算智能研究 institute）；Department of Computer Science and Artificial Intelligence (DECSAI), University of Granada（格拉纳达大学计算机科学与人工智能系）
MCL-AD: Multimodal Collaboration Learning for Zero-Shot 3D Anomaly Detection
Key Laboratory of Computing Power Network and Information Security, Ministry of Education, Shandong Computer Science Center (National Supercomputer Center in Jinan)（计算能力网络与信息安全部教育部重点实验室，山东计算机科学中心（济南国家超级计算机中心））；Qilu University of Technology (Shandong Academy of Sciences)（齐鲁工业大学（山东科学院））；Shandong Provincial Key Laboratory of Computing Power Internet and Service Computing, Shandong Fundamental Research Center for Computer Science（山东省计算能力互联网与服务计算重点实验室，山东省计算机科学基础研究中心）
TU Berlin（柏林工业大学）；Springer Heidelberg（施普林格·海德堡）；Princeton University（普林斯顿大学）；ABC Institute, Rupert-Karls-University Heidelberg（海德堡鲁珀特·卡尔斯大学ABC研究所）
The multi-source data generated by distributed systems, provide a holistic description of the system. Harnessing the joint distribution of the different modalities by a learning model can be beneficial for critical applications for maintenance of the distributed systems. One such important task is the task of anomaly detection where we are interested in detecting the deviation of the current behaviour of the system from the theoretically expected. In this work, we utilize the joint representation from the distributed traces and system log data for the task of anomaly detection in distributed s…
Out of Context: Reliability in Multimodal Anomaly Detection Requires Contextual Inference
Aalborg University（奥尔堡大学）；Pioneer Centre for Artificial Intelligence（先锋人工智能中心）；Technical University of Denmark（丹麦技术大学）；Milestone Systems（Milestone系统）；University of Bucharest（布加勒斯特大学）；University of Surrey（萨里大学）
Text-Attributed Graph Anomaly Detection via Multi-Scale Cross- and Uni-Modal Contrastive Learning
School of Computer Science and Technology, Xi'an Jiaotong University（西安交通大学计算机科学与技术学院）；Shaanxi Provincial Key Laboratory of Big Data Knowledge Engineering, Xi’an Jiaotong University（陕西省大数据知识工程重点实验室）；School of Distance Education, Xi’an Jiaotong University（西安交通大学继续教育学院）；University of Virginia（弗吉尼亚大学）
RWTH Aachen（亚琛工业大学）；LMU Munich（慕尼黑大学）；Humboldt-Universität zu Berlin（柏林洪堡大学）
Combining additive models and neural networks allows to broaden the scope of statistical regression and extend deep learning-based approaches by interpretable structured additive predictors at the same time. Existing attempts uniting the two modeling approaches are, however, limited to very specific combinations and, more importantly, involve an identifiability issue. As a consequence, interpretability and stable estimation are typically lost. We propose a general framework to combine structured regression models and deep neural networks into a unifying network architecture. To overcome the in…
Anomaly estimation, or the problem of finding a subset of a dataset that differs from the rest of the dataset, is a classic problem in machine learning and data mining. In both theoretical work and in applications, the anomaly is assumed to have a specific structure defined by membership in an $\textit{anomaly family}$. For example, in temporal data the anomaly family may be time intervals, while in network data the anomaly family may be connected subgraphs. The most prominent approach for anomaly estimation is to compute the Maximum Likelihood Estimator (MLE) of the anomaly; however, it was r…
Max Planck Institute for Informatics（马克斯·普朗克信息学研究所）；Bosch Center for Artificial Intelligence（博世人工智能中心）；CISPA Helmholtz Center for Information Security（CISPA亥姆霍兹信息安全中心）
Prediction of future states of the environment and interacting agents is a key competence required for autonomous agents to operate successfully in the real world. Prior work for structured sequence prediction based on latent variable models imposes a uni-modal standard Gaussian prior on the latent variables. This induces a strong model bias which makes it challenging to fully capture the multi-modality of the distribution of the future states. In this work, we introduce Conditional Flow Variational Autoencoders (CF-VAE) using our novel conditional normalizing flow based prior to capture compl…
Harbin Institute of Technology (Shenzhen)（哈尔滨工业大学（深圳））；Peng Cheng Laboratory（鹏城实验室）；Peking University（北京大学）
As two important textual modalities in electronic health records (EHR), both structured data (clinical codes) and unstructured data (clinical narratives) have recently been increasingly applied to the healthcare domain. Most existing EHR-oriented studies, however, either focus on a particular modality or integrate data from different modalities in a straightforward manner, which usually treats structured and unstructured data as two independent sources of information about patient admission and ignore the intrinsic interactions between them. In fact, the two modalities are documented during th…
Vision-Language Pretraining (VLP) and Foundation models have been the go-to recipe for achieving SoTA performance on general benchmarks. However, leveraging these powerful techniques for more complex vision-language tasks, such as cooking applications, with more structured input data, is still little investigated. In this work, we propose to leverage these techniques for structured-text based computational cuisine tasks. Our strategy, dubbed VLPCook, first transforms existing image-text pairs to image and structured-text pairs. This allows to pretrain our VLPCook model using VLP objectives ada…
KAIST（韩国科学技术院）；Seoul National University Bundang Hospital（首尔大学盆唐医院）
As the volume of Electronic Health Records (EHR) sharply grows, there has been emerging interest in learning the representation of EHR for healthcare applications. Representation learning of EHR requires appropriate modeling of the two dominant modalities in EHR: structured data and unstructured text. In this paper, we present MedGTX, a pre-trained model for multi-modal representation learning of the structured and textual EHR data. MedGTX uses a novel graph encoder to exploit the graphical nature of structured EHR data, and a text encoder to handle unstructured text, and a cross-modal encoder…
Western Washington University（西华盛顿大学）；Pacific Northwest National Laboratory（太平洋西北国家实验室）
Analysis of an organization's computer network activity is a key component of early detection and mitigation of insider threat, a growing concern for many organizations. Raw system logs are a prototypical example of streaming data that can quickly scale beyond the cognitive power of a human analyst. As a prospective filter for the human analyst, we present an online unsupervised deep learning approach to detect anomalous network activity from system logs in real time. Our models decompose anomaly scores into the contributions of individual user behavior features for increased interpretability …
Multi-Modal Dataset Creation for Federated Learning with DICOM Structured Reports
DZHK (German Centre for Cardiovascular Research, all partner sites)（德国心血管研究中心（所有合作站点））；Heidelberg University Hospital（海德堡大学医院）；Informatics for Life Institute（生命信息学研究所）；University Medical Center Hamburg-Eppendorf（汉堡埃彭多夫大学医学中心）；University Hospital Münster（明斯特大学医院）
