Improving Model Robustness with Latent Distribution Locally and Globally
Duke Kunshan University（昆山杜克大学）；University of Liverpool（利物浦大学）；Shanghai Artificial Intelligence Laboratory（上海人工智能实验室）；Xi’an Jiaotong-Liverpool University（西交利物浦大学）
In this work, we consider model robustness of deep neural networks against adversarial attacks from a global manifold perspective. Leveraging both the local and global latent information, we propose a novel adversarial training method through robust optimization, and a tractable way to generate Latent Manifold Adversarial Examples (LMAEs) via an adversarial game between a discriminator and a classifier. The proposed adversarial training with latent distribution (ATLD) method defends against adversarial attacks by crafting LMAEs with the latent manifold in an unsupervised manner. ATLD preserves the local and global information of latent manifold and promises improved robustness against adversarial attacks. To verify the effectiveness of our proposed method, we conduct extensive experiments over different datasets (e.g., CIFAR-10, CIFAR-100, SVHN) with different adversarial attacks (e.g., PGD, CW), and show that our method substantially outperforms the state-of-the-art (e.g., Feature Scattering) in adversarial robustness by a large accuracy margin. The source codes are available at https://github.com/LitterQ/ATLD-pytorch.
Adaptive gradient methods for stochastic optimization adjust the learning rate for each parameter locally. However, there is also a global learning rate which must be tuned in order to get the best performance. In this paper, we present a new algorithm that adapts the learning rate locally for each parameter separately, and also globally for all parameters together. Specifically, we modify Adam, a popular method for training deep learning models, with a coefficient that captures properties of the objective function. Empirically, we show that our method, which we call Eve, outperforms Adam and …
Despite the recent success of Graph Neural Networks (GNNs), training GNNs on large graphs remains challenging. The limited resource capacities of the existing servers, the dependency between nodes in a graph, and the privacy concern due to the centralized storage and model learning have spurred the need to design an effective distributed algorithm for GNN training. However, existing distributed GNN training methods impose either excessive communication costs or large memory overheads that hinders their scalability. To overcome these issues, we propose a communication-efficient distributed GNN …
The University of Liverpool（利物浦大学）；Japan Advanced Institute of Science and Technology（日本先进科学技术大学）；National Institute of Informatics（日本国立信息学研究所）；Japan Science and Technology Agency, ERATO, Kawarabayashi Large Graph Project（日本科学技术振兴机构 ERATO 河原林大型图项目）
Distributed word embeddings have shown superior performances in numerous Natural Language Processing (NLP) tasks. However, their performances vary significantly across different tasks, implying that the word embeddings learnt by those methods capture complementary aspects of lexical semantics. Therefore, we believe that it is important to combine the existing word embeddings to produce more accurate and complete \emph{meta-embeddings} of words. For this purpose, we propose an unsupervised locally linear meta-embedding learning method that takes pre-trained word embeddings as the input, and pro…
Forecasting high-dimensional time series plays a crucial role in many applications such as demand forecasting and financial predictions. Modern datasets can have millions of correlated time-series that evolve together, i.e they are extremely high dimensional (one dimension for each individual time-series). There is a need for exploiting global patterns and coupling them with local calibration for better prediction. However, most recent deep learning approaches in the literature are one-dimensional, i.e, even though they are trained on the whole dataset, during prediction, the future forecast f…
University of Freiburg（弗赖堡大学）；Google DeepMind（谷歌DeepMind）
We introduce Embed to Control (E2C), a method for model learning and control of non-linear dynamical systems from raw pixel images. E2C consists of a deep generative model, belonging to the family of variational autoencoders, that learns to generate image trajectories from a latent space in which the dynamics is constrained to be locally linear. Our model is derived directly from an optimal control formulation in latent space, supports long-term prediction of image sequences and exhibits strong performance on a variety of complex control problems.
Recent research on the time-domain audio separation networks (TasNets) has brought great success to speech separation. Nevertheless, conventional TasNets struggle to satisfy the memory and latency constraints in industrial applications. In this regard, we design a low-cost high-performance architecture, namely, globally attentive locally recurrent (GALR) network. Alike the dual-path RNN (DPRNN), we first split a feature sequence into 2D segments and then process the sequence along both the intra- and inter-segment dimensions. Our main innovation lies in that, on top of features recurrently pro…
Jiangsu University（江苏大学）；Nanjing University of Aeronautics and Astronautics (NUAA)（南京航空航天大学）
In multi-label learning, the issue of missing labels brings a major challenge. Many methods attempt to recovery missing labels by exploiting low-rank structure of label matrix. However, these methods just utilize global low-rank label structure, ignore both local low-rank label structures and label discriminant information to some extent, leaving room for further performance improvement. In this paper, we develop a simple yet effective discriminant multi-label learning (DM2L) method for multi-label learning with missing labels. Specifically, we impose the low-rank structures on all the predict…
University of Tokyo（东京大学）；University of Oregon（俄勒冈大学）；Columbia University（哥伦比亚大学）；University of Pittsburgh（匹兹堡大学）；Carnegie Mellon University（卡内基梅隆大学）
Federated learning is a method of training models on private data distributed over multiple devices. To keep device data private, the global model is trained by only communicating parameters and updates which poses scalability challenges for large models. To this end, we propose a new federated learning algorithm that jointly learns compact local representations on each device and a global model across all devices. As a result, the global model can be smaller since it only operates on local representations, reducing the number of communicated parameters. Theoretically, we provide a generalizat…
DeepMind；Google Research（谷歌研究）；Stanford University（斯坦福大学）；Facebook AI Research（Facebook人工智能研究院）；VinAI
Many real-world sequential decision-making problems can be formulated as optimal control with high-dimensional observations and unknown dynamics. A promising approach is to embed the high-dimensional observations into a lower-dimensional latent representation space, estimate the latent dynamics model, then utilize this model for control in the latent space. An important open question is how to learn a representation that is amenable to existing control algorithms? In this paper, we focus on learning representations for locally-linear control algorithms, such as iterative LQR (iLQR). By formula…
Learning Locally, Revising Globally: Global Reviser for Federated Learning with Noisy Labels
College of Computer Science, Sichuan University, Chengdu, China（四川大学计算机学院，中国成都）；Engineering Research Center of Machine Learning（机器学习工程研究中心）；University of Sydney, Sydney, Australia（悉尼大学，澳大利亚悉尼）；Southeast University, Nanjing, China（东南大学，中国南京）
Hong Kong Baptist University（香港浸会大学）；The Chinese University of Hong Kong（香港中文大学）
Regular medical records are useful for medical practitioners to analyze and monitor patient health status especially for those with chronic disease, but such records are usually incomplete due to unpunctuality and absence of patients. In order to resolve the missing data problem over time, tensor-based model is suggested for missing data imputation in recent papers because this approach makes use of low rank tensor assumption for highly correlated data. However, when the time intervals between records are long, the data correlation is not high along temporal direction and such assumption is no…
Stanford University（斯坦福大学）；VinAI；Google Research（谷歌研究院）；Facebook AI Research（Facebook人工智能研究院）
High-dimensional observations and unknown dynamics are major challenges when applying optimal control to many real-world decision making tasks. The Learning Controllable Embedding (LCE) framework addresses these challenges by embedding the observations into a lower dimensional latent space, estimating the latent dynamics, and then performing control directly in the latent space. To ensure the learned latent dynamics are predictive of next-observations, all existing LCE approaches decode back into the observation space and explicitly perform next-observation prediction---a challenging high-dime…
Learn Globally, Speak Locally: Bridging the Gaps in Multilingual Reasoning
Massachusetts Institute of Technology（麻省理工学院）；Havard Univeristy（哈佛大学）；LG CNS；Université de Montréal（蒙特利尔大学）；Mila（米拉研究所）；Google（谷歌）；Stanford University（斯坦福大学）
University of Geneva（日内瓦大学）；Geneva School of Business Administration, HES-SO（日内瓦高等商业学院（瑞士西部高等专业学院））；Apple（苹果公司）；Deepmind（深度思维）
Episodic and semantic memory are critical components of the human memory model. The theory of complementary learning systems (McClelland et al., 1995) suggests that the compressed representation produced by a serial event (episodic memory) is later restructured to build a more generalized form of reusable knowledge (semantic memory). In this work we develop a new principled Bayesian memory allocation scheme that bridges the gap between episodic and semantic memory via a hierarchical latent variable model. We take inspiration from traditional heap allocation and extend the idea of locally conti…
This paper introduces Attentive Implicit Representation Networks (AIR-Nets), a simple, but highly effective architecture for 3D reconstruction from point clouds. Since representing 3D shapes in a local and modular fashion increases generalization and reconstruction quality, AIR-Nets encode an input point cloud into a set of local latent vectors anchored in 3D space, which locally describe the object's geometry, as well as a global latent description, enforcing global consistency. Our model is the first grid-free, encoder-based approach that locally describes an implicit function. The vector at…
Attri-Net: A Globally and Locally Inherently Interpretable Model for Multi-Label Classification Using Class-Specific Counterfactuals
Cluster of Excellence: Machine Learning - New Perspectives for Science, University of Tübingen（机器学习新视角科学卓越中心，图宾根大学）；Department of Computer Science, Friedrich-Alexander-Universität Erlangen-Nürnberg（计算机科学系，埃尔兰根-纽伦堡弗里德里希-亚历山大大学）；Hertie Institute for Artificial Intelligence in Brain Health, University of Tübingen（脑健康人工智能研究所，图宾根大学）；Department of Diabetes, Endocrinology, Nutritional Medicine and Metabolism, Inselspital, Bern University Hospital, University of Bern（糖尿病、内分泌学、营养医学和代谢学系，伯恩大学医院，伯恩大学）；Faculty of Health Sciences and Medicine, University of Lucerne（健康科学与医学学院，卢塞恩大学）
