Knowledge-Embedded Latent Projection for Robust Representation Learning
Department of Statistics and Data Science, Carnegie Mellon University（卡内基梅隆大学统计学与数据科学系）；Department of Statistics, Columbia University（哥伦比亚大学统计学系）；Department of Neurology, University of Pittsburgh（匹兹堡大学神经病学系）；Department of Biostatistics, Harvard University（哈佛大学生物统计学系）
Latent space models are widely used for analyzing high-dimensional discrete data matrices, such as patient-feature matrices in electronic health records (EHRs), by capturing complex dependence structures through low-dimensional embeddings. However, estimation becomes challenging in the imbalanced regime, where one matrix dimension is much larger than the other. In EHR applications, cohort sizes are often limited by disease prevalence or data availability, whereas the feature space remains extremely large due to the breadth of medical coding system. Motivated by the increasing availability of external semantic embeddings, such as pre-trained embeddings of clinical concepts in EHRs, we propose a knowledge-embedded latent projection model that leverages semantic side information to regularize representation learning. Specifically, we model column embeddings as smooth functions of semantic embeddings via a mapping in a reproducing kernel Hilbert space. We develop a computationally efficient two-step estimation procedure that combines semantically guided subspace construction via kernel principal component analysis with scalable projected gradient descent. We establish estimation error bounds that characterize the trade-off between statistical error and approximation error induced by the kernel projection. Furthermore, we provide local convergence guarantees for our non-convex optimization procedure. Extensive simulation studies and a real-world EHR application demonstrate the effectiveness of the proposed method.
Institute of Electronic and Information Engineering of UESTC（电子科技大学电子信息工程研究院）；University of Electronic Science and Technology of China（电子科技大学）
With the representation learning capability of the deep learning models, deep embedded multi-view clustering (MVC) achieves impressive performance in many scenarios and has become increasingly popular in recent years. Although great progress has been made in this field, most existing methods merely focus on learning the latent representations and ignore that learning the latent graph of nodes also provides available information for the clustering task. To address this issue, in this paper we propose Deep Embedded Multi-view Clustering via Jointly Learning Latent Representations and Graphs (DMV…
Platonic Projection Structures: Operator-Induced Observability in Representation Learning
Department of Applied Information Engineering, Faculty of Engineering, Suwa University of Science（诹访理工大学工程学院应用信息工程系）；Department of Information Engineering, Kanazawa Gakuin University（金泽学院大学信息工程系）
In representation learning and non-linear dimension reduction, there is a huge interest to learn the 'disentangled' latent variables, where each sub-coordinate almost uniquely controls a facet of the observed data. While many regularization approaches have been proposed on variational autoencoders, heuristic tuning is required to balance between disentanglement and loss in reconstruction accuracy -- due to the unsupervised nature, there is no principled way to find an optimal weight for regularization. Motivated to completely bypass regularization, we consider a projection strategy: modifying …
Several regularization methods have recently been introduced which force the latent activations of an autoencoder or deep neural network to conform to either a Gaussian or hyperspherical distribution, or to minimize the implicit rank of the distribution in latent space. In the present work, we introduce a novel regularizing loss function which simulates a pairwise repulsive force between items and an attractive force of each item toward the origin. We show that minimizing this loss function in isolation achieves a hyperspherical distribution. Moreover, when used as a regularizing term, the sca…
Worcester Polytechnic Institute（伍斯特理工学院）；Beth Israel Deaconess Medical Center（贝斯以色列女执事医疗中心）；Massachusetts College of Pharmacy and Health Services（麻省药科与健康科学学院）
The analysis of clinical questionnaire data comes with many inherent challenges. These challenges include the handling of data with missing fields, as well as the overall interpretation of a dataset with many fields of different scales and forms. While numerous methods have been developed to address these challenges, they are often not robust, statistically sound, or easily interpretable. Here, we propose a latent factor modeling framework that extends the principal component analysis for both categorical and quantitative data with missing elements. The model simultaneously provides the princi…
German Center for Neurodegenerative Diseases（德国神经退行性疾病中心）；University of Cambridge（剑桥大学）
Linear projections are widely used in the analysis of high-dimensional data. In unsupervised settings where the data harbour latent classes/clusters, the question of whether class discriminatory signals are retained under projection is crucial. In the case of mean differences between classes, this question has been well studied. However, in many contemporary applications, notably in biomedicine, group differences at the level of covariance or graphical model structure are important. Motivated by such applications, in this paper we ask whether linear projections can preserve differences in seco…
Generative Adversarial networks (GANs) have obtained remarkable success in many unsupervised learning tasks and unarguably, clustering is an important unsupervised learning problem. While one can potentially exploit the latent-space back-projection in GANs to cluster, we demonstrate that the cluster structure is not retained in the GAN latent space. In this paper, we propose ClusterGAN as a new mechanism for clustering using GANs. By sampling latent variables from a mixture of one-hot encoded variables and continuous latent variables, coupled with an inverse network (which projects the data to…
Compressed sensing techniques enable efficient acquisition and recovery of sparse, high-dimensional data signals via low-dimensional projections. In this work, we propose Uncertainty Autoencoders, a learning framework for unsupervised representation learning inspired by compressed sensing. We treat the low-dimensional projections as noisy latent representations of an autoencoder and directly learn both the acquisition (i.e., encoding) and amortized recovery (i.e., decoding) procedures. Our learning objective optimizes for a tractable variational lower bound to the mutual information between th…
University of Aberdeen（阿伯丁大学）；University of Sheffield（谢菲尔德大学）；University of Surrey（萨里大学）
We tackle the problem disentangling the latent space of an autoencoder in order to separate labelled attribute information from other characteristic information. This then allows us to change selected attributes while preserving other information. Our method, matrix subspace projection, is much simpler than previous approaches to latent space factorisation, for example not requiring multiple discriminators or a careful weighting among their loss functions. Furthermore our new model can be applied to autoencoders as a plugin, and works across diverse domains such as images or text. We demonstra…
Okinawa Institute of Science and Technology（冲绳科学技术大学院大学）
Robot kinematics data, despite being a high dimensional process, is highly correlated, especially when considering motions grouped in certain primitives. These almost linear correlations within primitives allow us to interpret the motions as points drawn close to a union of low-dimensional linear subspaces in the space of all motions. Motivated by results of embedding theory, in particular, generalizations of Whitney embedding theorem, we show that random linear projection of motor sequences into low dimensional space loses very little information about structure of kinematics data. Projected …
Video Latent Flow Matching: Optimal Polynomial Projections for Video Interpolation and Extrapolation
Wyoming Seminary（怀俄明神学院）；Simons Institute for the Theory of Computing, University of California, Berkeley（加州大学伯克利分校西蒙斯理论计算研究所）；Sun Yat-sen University（中山大学）
本文提出 Video Latent Flow Matching (VLFM)，利用预训练图像生成模型和 HiPPO 最优多项式投影建模潜在块流，实现任意帧率的视频插值与外推生成，并具备理论逼近误差和时间尺度鲁棒性优势。
Deep Spectral Learning of Embedded Latent Transfer Operators for Stochastic Dynamical Systems
Graduate School of Information Science and Technology, The University of Osaka（大阪大学信息科学与技术研究生院）；Center for Advanced Intelligence Project, RIKEN（理化学研究所先进智能项目中心）
Counterfactual explanations represent the minimal change to a data sample that alters its predicted classification, typically from an unfavorable initial class to a desired target class. Counterfactuals help answer questions such as "what needs to change for this application to get accepted for a loan?". A number of recently proposed approaches to counterfactual generation give varying definitions of "plausible" counterfactuals and methods to generate them. However, many of these methods are computationally intensive and provide unconvincing explanations. Here we introduce SharpShooter, a meth…
Nagoya Institute of Technology（名古屋工业大学）；RIKEN Center for Advanced Intelligence Project（理化学研究所先进智能项目中心）
Here, we propose a new reconstruction method of smooth time-series signals. A key concept of this study is not considering the model in signal space, but in delay-embedded space. In other words, we indirectly represent a time-series signal as an output of inverse delay-embedding of a matrix, and the matrix is constrained. Based on the model under inverse delay-embedding, we propose to constrain the matrix to be rank-1 with smooth factor vectors. The proposed model is closely related to the convolutional model, and quadratic variation (QV) regularization. Especially, the proposed method can be …
ESAT-STADIUS；KU Leuven（鲁汶大学）；Department of Computer Science（计算机科学系）；University of Toronto（多伦多大学）
Neural ordinary differential equations (Neural ODEs) are an effective framework for learning dynamical systems from irregularly sampled time series data. These models provide a continuous-time latent representation of the underlying dynamical system where new observations at arbitrary time points can be used to update the latent representation of the dynamical system. Existing parameterizations for the dynamics functions of Neural ODEs limit the ability of the model to retain global information about the time series; specifically, a piece-wise integration of the latent process between observat…
Cross-Domain Few-Shot Learning with Coalescent Projections and Latent Space Reservation
STEM, University of South Australia（南澳大利亚大学STEM学院）；Systems Research Institute, Polish Academy of Sciences（波兰科学院系统研究所）；Departement of Informatics, Institut Teknologi Sepuluh Nopember（十月份十号技术大学信息学院）
By illuminating latent structures in a corpus of text, topic models are an essential tool for categorizing, summarizing, and exploring large collections of documents. Probabilistic topic models, such as latent Dirichlet allocation (LDA), describe how words in documents are generated via a set of latent distributions called topics. Recently, the Embedded Topic Model (ETM) has extended LDA to utilize the semantic information in word embeddings to derive semantically richer topics. As LDA and its extensions are unsupervised models, they aren't defined to make efficient use of a user's prior knowl…
Probabilistic Reduced-Dimensional Vector Autoregressive Modeling with Oblique Projections
Hong Kong Institute for Data Science, City University of Hong Kong, Hong Kong（香港数据科学研究所，香港城市大学，香港）；The Institute of Data Science, Lingnan University, Hong Kong（数据科学研究所，岭南大学，香港）
Data Obfuscation through Latent Space Projection (LSP) for Privacy-Preserving AI Governance: Case Studies in Medical Diagnosis and Finance Fraud Detection
College of Information and Computer Sciences（信息与计算机科学学院）；University of Massachusetts Amherst（马萨诸塞大学阿默斯特分校）
Complex textual information extraction tasks are often posed as sequence labeling or \emph{shallow parsing}, where fields are extracted using local labels made consistent through probabilistic inference in a graphical model with constrained transitions. Recently, it has become common to locally parametrize these models using rich features extracted by recurrent neural networks (such as LSTM), while enforcing consistent outputs through a simple linear-chain model, representing Markovian dependencies between successive labels. However, the simple graphical model structure belies the often comple…
Center for Energy and Geo Processing (CeGP)（能源与地理处理中心（CeGP））；King Fahd University of Petroleum and Engineering (KFUPM)（法赫德国王石油与工程大学（KFUPM））
We developed two machine learning frameworks that could assist in automated litho-stratigraphic interpretation of seismic volumes without any manual hand labeling from an experienced seismic interpreter. The first framework is an unsupervised hierarchical clustering model to divide seismic images from a volume into certain number of clusters determined by the algorithm. The clustering framework uses a combination of density and hierarchical techniques to determine the size and homogeneity of the clusters. The second framework consists of a self-supervised deep learning framework to label regio…
Stanford University（斯坦福大学）；University of California San Diego（加州大学圣地亚哥分校）
Word embedding models such as GloVe are widely used in natural language processing (NLP) research to convert words into vectors. Here, we provide a preliminary guide to probe latent emotions in text through GloVe word vectors. First, we trained a neural network model to predict continuous emotion valence ratings by taking linguistic inputs from Stanford Emotional Narratives Dataset (SEND). After interpreting the weights in the model, we found that only a few dimensions of the word vectors contributed to expressing emotions in text, and words were clustered on the basis of their emotional polar…
Embedded Variational Neural Stochastic Differential Equations for Learning Heterogeneous Dynamics
National Institute of Technology, Rourkela, India（印度国立理工学院鲁尔克拉分校）；Poverty Alleviation Research Centre, National Institute of Technology, Rourkela, India（印度国立理工学院鲁尔克拉分校扶贫研究中心）
University of Quebec at Montreal（蒙特利尔魁北克大学）；University of Windsor（温莎大学）
Clustering using deep autoencoders has been thoroughly investigated in recent years. Current approaches rely on simultaneously learning embedded features and clustering the data points in the latent space. Although numerous deep clustering approaches outperform the shallow models in achieving favorable results on several high-semantic datasets, a critical weakness of such models has been overlooked. In the absence of concrete supervisory signals, the embedded clustering objective function may distort the latent space by learning from unreliable pseudo-labels. Thus, the network can learn non-re…
Jilin University（吉林大学）；University of Tokyo（东京大学）；Japan Advanced Institute of Science and Technology（日本北陆先端科学技术大学院大学）
We propose an interactive editing method that allows humans to help deep neural networks (DNNs) learn a latent space more consistent with human knowledge, thereby improving classification accuracy on indistinguishable ambiguous data. Firstly, we visualize high-dimensional data features through dimensionality reduction methods and design an interactive system \textit{SpaceEditing} to display the visualized data. \textit{SpaceEditing} provides a 2D workspace based on the idea of spatial layout. In this workspace, the user can move the projection data in it according to the system guidance. Then,…
Training deep neural networks at the edge on light computational devices, embedded systems and robotic platforms is nowadays very challenging. Continual learning techniques, where complex models are incrementally trained on small batches of new data, can make the learning problem tractable even for CPU-only embedded devices enabling remarkable levels of adaptiveness and autonomy. However, a number of practical problems need to be solved: catastrophic forgetting before anything else. In this paper we introduce an original technique named "Latent Replay" where, instead of storing a portion of pa…
Deep Embedded Multiplicative DMD for Algebra-Preserving Koopman Learning
Department of Mathematics, Imperial College London（帝国理工学院数学系）；Department of Applied Mathematics and Theoretical Physics, University of Cambridge（剑桥大学应用数学与理论物理系）
Decoding Latent Spaces: Assessing the Interpretability of Time Series Foundation Models for Visual Analytics
Computer Systems Engineering Dpt., Universidad Politécnica de Madrid, Madrid (Spain)（马德里理工大学计算机系统工程系）；Department of Computer Science and Engineering, Seoul National University of Science and Technology, Seoul (Korea)（首尔科学技术大学计算机科学与工程系）
本研究评估了时间序列基础模型 MOMENT 在可视化分析任务中潜在空间的解释性，发现微调虽降低了损失但嵌入空间可解释性改善有限，指出基础模型仍需方法改进但显著提升了交互分析效率。
Institute for Infocomm Research, Agency for Science, Technology and Research (A*STAR)（信息通信研究院，新加坡科技研究局（A*STAR））；Center for Data Sciences, International Institute of Information Technology Bangalore (IIITB)（班加罗尔国际信息技术学院数据科学中心）；School of Computer Science and Engineering, Nanyang Technological University（南洋理工大学计算机科学与工程学院）；A*STAR Centre for Frontier AI Research（新加坡科技研究局前沿人工智能研究中心）
Neighborhood-Adaptive Generalized Linear Graph Embedding with Latent Pattern Mining
School of Computer and Information, Anhui Normal University（安徽师范大学计算机与信息学院）；Anhui Provincial Key Laboratory of Industrial Intelligent Data Security（安徽省工业智能数据安全重点实验室）；Institute of Automation, State Key Laboratory of Multimodal Artificial Intelligence Systems, Chinese Academy of Sciences（自动化研究所，多模态人工智能系统国家重点实验室，中国科学院）；School of Artificial Intelligence, University of Chinese Academy of Sciences（中国科学院大学人工智能学院）
Geodesic Calculus on Implicitly Defined Latent Manifolds
University of Bonn, Germany（波恩大学，德国）；Université Paris–Saclay, ENS Paris–Saclay, Centre Borelli, France（巴黎-萨克莱大学，巴黎-萨克莱高等师范学院，Borelli研究中心，法国）；University of Münster, Germany（穆尔斯特大学，德国）
University of South Carolina（南卡罗来纳大学）；University of North Carolina at Charlotte（北卡罗来纳大学夏洛特分校）
Real-time nonlinear Bayesian filtering algorithms are overwhelmed by data volume, velocity and increasing complexity of computational models. In this paper, we propose a novel ensemble based nonlinear Bayesian filtering approach which only requires a small number of simulations and can be applied to high-dimensional systems in the presence of intractable likelihood functions. The proposed approach uses linear latent projections to estimate the joint probability distribution between states, parameters, and observables using a mixture of Gaussian components generated by the reconstruction error …
Lawrence Livermore National Laboratory（劳伦斯利弗莫尔国家实验室）
In the past few years, Generative Adversarial Networks (GANs) have dramatically advanced our ability to represent and parameterize high-dimensional, non-linear image manifolds. As a result, they have been widely adopted across a variety of applications, ranging from challenging inverse problems like image completion, to problems such as anomaly detection and adversarial defense. A recurring theme in many of these applications is the notion of projecting an image observation onto the manifold that is inferred by the generator. In this context, Projected Gradient Descent (PGD) has been the most …
