Although deep learning models have achieved state-of-the-art performance on a number of vision tasks, generalization over high dimensional multi-modal data, and reliable predictive uncertainty estimation are still active areas of research. Bayesian approaches including Bayesian Neural Nets (BNNs) do not scale well to modern computer vision tasks, as they are difficult to train, and have poor generalization under dataset-shift. This motivates the need for effective ensembles which can generalize and give reliable uncertainty estimates. In this paper, we target the problem of generating effective ensembles of neural networks by encouraging diversity in prediction. We explicitly optimize a diversity inducing adversarial loss for learning the stochastic latent variables and thereby obtain diversity in the output predictions necessary for modeling multi-modal data. We evaluate our method on benchmark datasets: MNIST, CIFAR100, TinyImageNet and MIT Places 2, and compared to the most competitive baselines show significant improvements in classification accuracy, under a shift in the data distribution and in out-of-distribution detection. Code will be released in this url https://github.com/rvl-lab-utoronto/dibs
Michigan State University（密歇根州立大学）；Argonne National Laboratory（阿贡国家实验室）；Google Research（谷歌研究院）
The information bottleneck framework provides a systematic approach to learning representations that compress nuisance information in the input and extract semantically meaningful information about predictions. However, the choice of a prior distribution that fixes the dimensionality across all the data can restrict the flexibility of this approach for learning robust representations. We present a novel sparsity-inducing spike-slab categorical prior that uses sparsity as a mechanism to provide the flexibility that allows each data point to learn its own dimension distribution. In addition, it …
Sparse Gaussian processes and various extensions thereof are enabled through inducing points, that simultaneously bottleneck the predictive capacity and act as the main contributor towards model complexity. However, the number of inducing points is generally not associated with uncertainty which prevents us from applying the apparatus of Bayesian reasoning for identifying an appropriate trade-off. In this work we place a point process prior on the inducing points and approximate the associated posterior through stochastic variational inference. By letting the prior encourage a moderate number …
Sparse Gaussian Processes are a key component of high-throughput Bayesian Optimisation (BO) loops; however, we show that existing methods for allocating their inducing points severely hamper optimisation performance. By exploiting the quality-diversity decomposition of Determinantal Point Processes, we propose the first inducing point allocation strategy designed specifically for use in BO. Unlike existing methods which seek only to reduce global uncertainty in the objective function, our approach provides the local high-fidelity modelling of promising regions required for precise optimisation…
Latent Variable Models (LVMs) are a large family of machine learning models providing a principled and effective way to extract underlying patterns, structure and knowledge from observed data. Due to the dramatic growth of volume and complexity of data, several new challenges have emerged and cannot be effectively addressed by existing LVMs: (1) How to capture long-tail patterns that carry crucial information when the popularity of patterns is distributed in a power-law fashion? (2) How to reduce model complexity and computational cost without compromising the modeling power of LVMs? (3) How t…
Standard reinforcement learning methods aim to master one way of solving a task whereas there may exist multiple near-optimal policies. Being able to identify this collection of near-optimal policies can allow a domain expert to efficiently explore the space of reasonable solutions. Unfortunately, existing approaches that quantify uncertainty over policies are not ultimately relevant to finding policies with qualitatively distinct behaviors. In this work, we formalize the difference between policies as a difference between the distribution of trajectories induced by each policy, which encourag…
Recently diversity-inducing regularization methods for latent variable models (LVMs), which encourage the components in LVMs to be diverse, have been studied to address several issues involved in latent variable modeling: (1) how to capture long-tail patterns underlying data; (2) how to reduce model complexity without sacrificing expressivity; (3) how to improve the interpretability of learned patterns. While the effectiveness of diversity-inducing regularizers such as the mutual angular regularizer has been demonstrated empirically, a rigorous theoretical analysis of them is still missing. In…
Deep ensembles perform better than a single network thanks to the diversity among their members. Recent approaches regularize predictions to increase diversity; however, they also drastically decrease individual members' performances. In this paper, we argue that learning strategies for deep ensembles need to tackle the trade-off between ensemble diversity and individual accuracies. Motivated by arguments from information theory and leveraging recent advances in neural estimation of conditional mutual information, we introduce a novel training criterion called DICE: it increases diversity by r…
Surveying the Effects of Quality, Diversity, and Complexity in Synthetic Data From Large Language Models
Georgia Tech（佐治亚理工学院）；Cornell University（康奈尔大学）；University of Limerick（利默里克大学）；Sakana AI；SynthLabs；Reka AI；Stanford University（斯坦福大学）；Eleuther AI；IBM；Cohere for AI Community；University of Bologna（博洛尼亚大学）；Cognizant AI Labs
Duke University（杜克大学）；Radiance Technologies（辐射技术公司）
Recent research finds CNN models for image classification demonstrate overlapped adversarial vulnerabilities: adversarial attacks can mislead CNN models with small perturbations, which can effectively transfer between different models trained on the same dataset. Adversarial training, as a general robustness improvement technique, eliminates the vulnerability in a single model by forcing it to learn robust features. The process is hard, often requires models with large capacity, and suffers from significant loss on clean data accuracy. Alternatively, ensemble methods are proposed to induce sub…
REALITrees: Rashomon Ensemble Active Learning for Interpretable Trees
Department of Statistics, University of Washington（统计学系，华盛顿大学）；Department of Computer Science, Duke University（计算机科学系，杜克大学）；Department of Sociology, University of Washington（社会学系，华盛顿大学）
Influence-Directed Distillation: Solving the Diversity Bottleneck in Sampled-Token On-Policy Distillation
University of North Carolina at Chapel Hill（北卡罗来纳大学教堂山分校）；University of Science and Technology of China（中国科学技术大学）；Shanghai University of Finance and Economics（上海财经大学）
It is widely known in the machine learning community that class noise can be (and often is) detrimental to inducing a model of the data. Many current approaches use a single, often biased, measurement to determine if an instance is noisy. A biased measure may work well on certain data sets, but it can also be less effective on a broader set of data sets. In this paper, we present noise identification using classifier diversity (NICD) -- a method for deriving a less biased noise measurement and integrating it into the learning process. To lessen the bias of the noise measure, NICD selects a div…
The ability to discover useful behaviours from past experience and transfer them to new tasks is considered a core component of natural embodied intelligence. Inspired by neuroscience, discovering behaviours that switch at bottleneck states have been long sought after for inducing plans of minimum description length across tasks. Prior approaches have either only supported online, on-policy, bottleneck state discovery, limiting sample-efficiency, or discrete state-action domains, restricting applicability. To address this, we introduce Model-Based Offline Options (MO2), an offline hindsight fr…
Department of Electrical Engineering and Computer Science, Pennsylvania State University, PA, USA（电气工程与计算机科学系，宾夕法尼亚州立大学）；College of Information Sciences and Technology, Pennsylvania State University, PA, USA（信息科学与技术学院，宾夕法尼亚州立大学）
DDTime: Dataset Distillation with Spectral Alignment and Information Bottleneck for Time-Series Forecasting
The City University of New York, CUNY（纽约城市大学）；Institute of Computing Technology, Chinese Academy of Sciences（中国科学院计算技术研究所）；Zhejiang University（浙江大学）；University of Illinois at Chicago（伊利诺伊大学香槟分校）；Tongji University（同济大学）
The Hong Kong University of Science and Technology（香港科技大学）；Hong Kong Polytechnic University（香港理工大学）
This paper investigates task-oriented communication for edge inference, where a low-end edge device transmits the extracted feature vector of a local data sample to a powerful edge server for processing. It is critical to encode the data into an informative and compact representation for low-latency inference given the limited bandwidth. We propose a learning-based communication scheme that jointly optimizes feature extraction, source coding, and channel coding in a task-oriented manner, i.e., targeting the downstream inference task rather than data reconstruction. Specifically, we leverage an…
The diversity and quantity of data warehouses, gathering data from distributed devices such as mobile devices, can enhance the success and robustness of machine learning algorithms. Federated learning enables distributed participants to collaboratively learn a commonly-shared model while holding data locally. However, it is also faced with expensive communication and limitations due to the heterogeneity of distributed data sources and lack of access to global data. In this paper, we investigate a practical distributed learning scenario where multiple downstream tasks (e.g., classifiers) could …
Token Communication in the Era of Large Models: An Information Bottleneck-Based Approach
State Key Laboratory of Networking and Switching Technology, Beijing University of Posts and Telecommunications（网络与交换技术国家重点实验室，北京邮电大学）；Department of Mathematics and Theories, Peng Cheng Laboratory（数学与理论部，鹏城实验室）；Department of Electronic Engineering, Tsinghua University（电子工程系，清华大学）；Pervasive Communications Center, Purple Mountain Laboratories（感知计算中心，紫金山实验室）；College of Computing and Data Science, Nanyang Technological University（计算与数据科学学院，南洋理工大学）
Fidelity, Diversity, and Privacy: A Multi-Dimensional LLM Evaluation for Clinical Data Augmentation
Universidad Politécnica de Madrid（马德里理工大学）；University Hospital Rey Juan Carlos（雷伊·卡洛斯大学医院）；University Hospital Jimenez Diaz Foundation（吉梅尼兹·迪亚兹基金会大学医院）；General Hospital of Villalba（维拉尔巴综合医院）；University Hospital Infanta Elena（伊菲塔·埃莱娜大学医院）；Universidad Catolica del Maule（马乌尔天主教大学）；Madrid Autonomous University（马德里自治大学）；CIBERSAM, ISCIII
Fully convolutional networks have shown outstanding performance in the salient object detection (SOD) field. The state-of-the-art (SOTA) methods have a tendency to become deeper and more complex, which easily homogenize their learned deep features, resulting in a clear performance bottleneck. In sharp contrast to the conventional ``deeper'' schemes, this paper proposes a ``wider'' network architecture which consists of parallel sub networks with totally different network architectures. In this way, those deep features obtained via these two sub networks will exhibit large diversity, which will…
Unsupervised skill discovery drives intelligent agents to explore the unknown environment without task-specific reward signal, and the agents acquire various skills which may be useful when the agents adapt to new tasks. In this paper, we propose "Multi-agent Skill Discovery"(MASD), a method for discovering skills for coordination patterns of multiple agents. The proposed method aims to maximize the mutual information between a latent code Z representing skills and the combination of the states of all agents. Meanwhile it suppresses the empowerment of Z on the state of any single agent by adve…
Value Decomposition (VD) aims to deduce the contributions of agents for decentralized policies in the presence of only global rewards, and has recently emerged as a powerful credit assignment paradigm for tackling cooperative Multi-Agent Reinforcement Learning (MARL) problems. One of the main challenges in VD is to promote diverse behaviors among agents, while existing methods directly encourage the diversity of learned agent networks with various strategies. However, we argue that these dedicated designs for agent networks are still limited by the indistinguishable VD network, leading to homo…
