首页/论文/2003.04286

arXiv 2003.042862020-03stat.MLcs.CVcs.LGcs.NE

Manifold Regularization for Locally Stable Deep Neural Networks

CSAIL（麻省理工学院计算机科学与人工智能实验室）；MIT（麻省理工学院）

Charles Jin, Martin Rinard

英文摘要：

We apply concepts from manifold regularization to develop new regularization techniques for training locally stable deep neural networks. Our regularizers are based on a sparsification of the graph Laplacian which holds with high probability when the data is sparse in high dimensions, as is common in deep learning. Empirically, our networks exhibit stability in a diverse set of perturbation models, including $\ell_2$, $\ell_\infty$, and Wasserstein-based perturbations; in particular, we achieve 40% adversarial accuracy on CIFAR-10 against an adaptive PGD attack using $\ell_\infty$ perturbations of size $ε= 8/255$, and state-of-the-art verified accuracy of 21% in the same perturbation model. Furthermore, our techniques are efficient, incurring overhead on par with two additional parallel forward passes through the network.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。

阅读原文

阅读 PDF ↗

arXiv 页面 ↗HTML 原文 ↗

本文内容

英文摘要相关论文

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。

继续探索

相关论文

A Distribution Dependent and Independent Complexity Analysis of Manifold Regularization

Delft University of Technology（代尔夫特理工大学）；University of Copenhagen（哥本哈根大学）

Manifold regularization is a commonly used technique in semi-supervised learning. It enforces the classification rule to be smooth with respect to the data-manifold. Here, we derive sample complexity bounds based on pseudo-dimension for models that add a convex data dependent regularization term to a supervised learning process, as is in particular done in Manifold regularization. We then compare the bound for those semi-supervised methods to purely supervised methods, and discuss a setting in which the semi-supervised method can only have a constant improvement, ignoring logarithmic terms. By…

2019-11-27cs.LGstat.ML

Avoiding unwanted results in locally linear embedding: A new understanding of regularization

National Sun Yat-sen University（国立中山大学）

We demonstrate that locally linear embedding (LLE) inherently admits some unwanted results when no regularization is used, even for cases in which regularization is not supposed to be needed in the original algorithm. The existence of one special type of result, which we call ``projection pattern'', is mathematically proved in the situation that an exact local linear relation is achieved in each neighborhood of the data. These special patterns as well as some other bizarre results that may occur in more general situations are shown by numerical examples on the Swiss roll with a hole embedded i…

2021-08-28cs.LGcs.NAmath.NA

Manifold regularization with GANs for semi-supervised learning

Institute for Infocomm Research, A*STAR（新加坡科技研究局资讯通信研究院）

Generative Adversarial Networks are powerful generative models that are able to model the manifold of natural images. We leverage this property to perform manifold regularization by approximating a variant of the Laplacian norm using a Monte Carlo approximation that is easily computed with the GAN. When incorporated into the semi-supervised feature-matching GAN we achieve state-of-the-art results for GAN-based semi-supervised learning on CIFAR-10 and SVHN benchmarks, with a method that is significantly easier to implement than competing methods. We also find that manifold regularization improv…

2018-07-11cs.LGstat.ML

Semi-Supervised Learning with GANs: Revisiting Manifold Regularization

Télécom ParisTech（巴黎高等电信学院）；Institute for Infocomm Research（信息通信研究院）；CentraleSupélec（中央理工-高等电力学院）；Nanyang Technological University（南洋理工大学）

GANS are powerful generative models that are able to model the manifold of natural images. We leverage this property to perform manifold regularization by approximating the Laplacian norm using a Monte Carlo approximation that is easily computed with the GAN. When incorporated into the feature-matching GAN of Improved GAN, we achieve state-of-the-art results for GAN-based semi-supervised learning on the CIFAR-10 dataset, with a method that is significantly easier to implement than competing methods.

2018-05-23cs.LGstat.ML

Random Manifold Sampling and Joint Sparse Regularization for Multi-label Feature Selection

Northeastern University（东北大学）；Shandong University（山东大学）

Multi-label learning is usually used to mine the correlation between features and labels, and feature selection can retain as much information as possible through a small number of features. $\ell_{2,1}$ regularization method can get sparse coefficient matrix, but it can not solve multicollinearity problem effectively. The model proposed in this paper can obtain the most relevant few features by solving the joint constrained optimization problems of $\ell_{2,1}$ and $\ell_{F}$ regularization.In manifold regularization, we implement random walk strategy based on joint information matrix, and ge…

2023-03-30cs.LGstat.ML

Locally Linear Image Structural Embedding for Image Structure Manifold Learning

University of Waterloo（滑铁卢大学）

Most of existing manifold learning methods rely on Mean Squared Error (MSE) or $\ell_2$ norm. However, for the problem of image quality assessment, these are not promising measure. In this paper, we introduce the concept of an image structure manifold which captures image structure features and discriminates image distortions. We propose a new manifold learning method, Locally Linear Image Structural Embedding (LLISE), and kernel LLISE for learning this manifold. The LLISE is inspired by Locally Linear Embedding (LLE) but uses SSIM rather than MSE. This paper builds a bridge between manifold l…

2019-08-25cs.CVcs.LGeess.IV

更多相关论文收起更多论文

基于Nyström型子抽样的流形正则化

Manifold regularization based on Nystr{ö}m type subsampling

Indian Institute of Technology Delhi（印度德里理工学院）

本文提出基于Nyström型子抽样的多惩罚正则化方法以降低大规模核方法的计算复杂度，给出向量值函数设置下的最优收敛速率理论分析，并在Caltech-101和NSL-KDD数据集上验证了其在图像分类与入侵检测中的有效性。

2017-10-13cs.LGstat.ML

基于改进的Diffusion Map的流形正则化分类模型

Manifold Regularization Classification Model Based On Improved Diffusion Map

Dalian University of Technology（大连理工大学）；Leceister Institution（莱斯特机构）

本文针对流形正则化模型局限于局部区域的问题，提出基于标签传播模型的改进方法，通过优化Diffusion Map的概率转移矩阵构建标签传播函数，并将其扩展至整个数据流形以作为分类器，实验验证了该改进模型的优越性。

2024-03-24cs.LGmath.OCstat.ML

Bayesian Manifold Learning: The Locally Linear Latent Variable Model (LL-LVM)

We introduce the Locally Linear Latent Variable Model (LL-LVM), a probabilistic model for non-linear manifold discovery that describes a joint distribution over observations, their manifold coordinates and locally linear maps conditioned on a set of neighbourhood relationships. The model allows straightforward variational optimisation of the posterior distribution on coordinates and locally linear maps from the latent space to the observation space given the data. Thus, the LL-LVM encapsulates the local-geometry preserving intuitions that underlie non-probabilistic methods such as locally line…

2015-12-01stat.ML

Causal Learning via Manifold Regularization

This paper frames causal structure estimation as a machine learning task. The idea is to treat indicators of causal relationships between variables as `labels' and to exploit available data on the variables of interest to provide features for the labelling task. Background scientific knowledge or any available interventional data provide labels on some causal relationships and the remainder are treated as unlabelled. To illustrate the key ideas, we develop a distance-based approach (based on bivariate histograms) within a manifold regularization framework. We present empirical results on three…

2019-08-29stat.ML

An Inexact Manifold Augmented Lagrangian Method for Adaptive Sparse Canonical Correlation Analysis with Trace Lasso Regularization

Canonical correlation analysis (CCA for short) describes the relationship between two sets of variables by finding some linear combinations of these variables that maximizing the correlation coefficient. However, in high-dimensional settings where the number of variables exceeds sample size, or in the case of that the variables are highly correlated, the traditional CCA is no longer appropriate. In this paper, an adaptive sparse version of CCA (ASCCA for short) is proposed by using the trace Lasso regularization. The proposed ASCCA reduces the instability of the estimator when the covariates a…

2020-03-20math.OCstat.ML