While neural network binary classifiers are often evaluated on metrics such as Accuracy and $F_1$-Score, they are commonly trained with a cross-entropy objective. How can this training-evaluation gap be addressed? While specific techniques have been adopted to optimize certain confusion matrix based metrics, it is challenging or impossible in some cases to generalize the techniques to other metrics. Adversarial learning approaches have also been proposed to optimize networks via confusion matrix based metrics, but they tend to be much slower than common training methods. In this work, we propose a unifying approach to training neural network binary classifiers that combines a differentiable approximation of the Heaviside function with a probabilistic view of the typical confusion matrix values using soft sets. Our theoretical analysis shows the benefit of using our method to optimize for a given evaluation metric, such as $F_1$-Score, with soft sets, and our extensive experiments show the effectiveness of our approach in several domains.
In this paper, we study the problem of unifying knowledge from a set of classifiers with different architectures and target classes into a single classifier, given only a generic set of unlabelled data. We call this problem Unifying Heterogeneous Classifiers (UHC). This problem is motivated by scenarios where data is collected from multiple sources, but the sources cannot share their data, e.g., due to privacy concerns, and only privately trained models can be shared. In addition, each source may not be able to gather data to train all classes due to data availability at each source, and may n…
In this draft, which reports on work in progress, we 1) adapt the information bottleneck functional by replacing the compression term by class-conditional compression, 2) relax this functional using a variational bound related to class-conditional disentanglement, 3) consider this functional as a training objective for stochastic neural networks, and 4) show that the latent representations are learned such that they can be used in a naive Bayes classifier. We continue by suggesting a series of experiments along the lines of Nonlinear In-formation Bottleneck [Kolchinsky et al., 2018], Deep Vari…
Rutgers, The State University of New Jersey（罗格斯新泽西州立大学）；University of South California（南加州大学）；University of California, Los Angeles（加州大学洛杉矶分校）
Asymmetric binary classification problems, in which the type I and II errors have unequal severity, are ubiquitous in real-world applications. To handle such asymmetry, researchers have developed the cost-sensitive and Neyman-Pearson paradigms for training classifiers to control the more severe type of classification error, say the type I error. The cost-sensitive paradigm is widely used and has straightforward implementations that do not require sample splitting; however, it demands an explicit specification of the costs of the type I and II errors, and an open question is what specification …
Bridging the Gap: Addressing Discrepancies in Diffusion Model Training for Classifier-Free Guidance
University of California, Los Angeles（加州大学洛杉矶分校）；Swiss Data Science Center, ETH Zürich（苏黎世联邦理工学院瑞士数据科学中心）；Swiss Data Science Center, Paul Scherrer Institute（保罗·谢勒研究所瑞士数据科学中心）
National University of Singapore（新加坡国立大学）；Technical University of Munich（慕尼黑工业大学）
The current state-of-the-art defense methods against adversarial examples typically focus on improving either empirical or certified robustness. Among them, adversarially trained (AT) models produce empirical state-of-the-art defense against adversarial examples without providing any robustness guarantees for large classifiers or higher-dimensional inputs. In contrast, existing randomized smoothing based models achieve state-of-the-art certified robustness while significantly degrading the empirical robustness against adversarial examples. In this paper, we propose a novel method, called \emph…
Causal Fair Metric: Bridging Causality, Individual Fairness, and Adversarial Robustness
Max Planck Institute for Intelligent Systems（马克斯·普朗克智能系统研究所）；Tübingen AI Center（蒂宾根人工智能中心）；McGill University（麦吉尔大学）；University of Montréal（蒙特利尔大学）；MILA（米拉计算科学研究所）
Multi-domain learning aims to benefit from simultaneously learning across several different but related domains. In this chapter, we propose a single framework that unifies multi-domain learning (MDL) and the related but better studied area of multi-task learning (MTL). By exploiting the concept of a \emph{semantic descriptor} we show how our framework encompasses various classic and recent MDL/MTL algorithms as special cases with different semantic descriptor encodings. As a second contribution, we present a higher order generalisation of this framework, capable of simultaneous multi-task-mul…
Generative Example-Based Explanations: Bridging the Gap between Generative Modeling and Explainability
Center for Artificial Intelligence and Robotics（人工智能与机器人中心）；Technical University of Applied Sciences Würzburg-Schweinfurt（应用科学大学魏玛-施维林分校）；Bielefeld University（比勒菲尔德大学）
University of California, Santa Barbara（加州大学圣巴巴拉分校）；Korea University（高丽大学）；Meta AI；Google Research（谷歌研究院）；Apple（苹果公司）
Building dense retrievers requires a series of standard procedures, including training and validating neural models and creating indexes for efficient search. However, these procedures are often misaligned in that training objectives do not exactly reflect the retrieval scenario at inference time. In this paper, we explore how the gap between training and inference in dense retrieval can be reduced, focusing on dense phrase retrieval (Lee et al., 2021) where billions of representations are indexed at inference. Since validating every dense retriever with a large-scale index is practically infe…
University of Toronto（多伦多大学）；Vector Institute（向量研究所）；DeepMind（深度思维）；Google Research（谷歌研究院）；Mila（米拉研究院）
We demonstrate that the choice of optimizer, neural network architecture, and regularizer significantly affect the adversarial robustness of linear neural networks, providing guarantees without the need for adversarial training. To this end, we revisit a known result linking maximally robust classifiers and minimum norm solutions, and combine it with recent results on the implicit bias of optimizers. First, we show that, under certain conditions, it is possible to achieve both perfect standard accuracy and a certain degree of robustness, simply by training an overparametrized model using the i…
Generative adversarial networks (GANs) are a framework that learns a generative distribution through adversarial training. Recently, their class-conditional extensions (e.g., conditional GAN (cGAN) and auxiliary classifier GAN (AC-GAN)) have attracted much attention owing to their ability to learn the disentangled representations and to improve the training stability. However, their training requires the availability of large-scale accurate class-labeled data, which are often laborious or impractical to collect in a real-world scenario. To remedy this, we propose a novel family of GANs called …
Bridging Generative and Discriminative Noisy-Label Learning via Direction-Agnostic EM Formulation
Cornell Tech（康奈尔科技）；Cornell University（康奈尔大学）；Department of Radiology, Stanford University（斯坦福大学放射科）；Oxford University（牛津大学）；Centre for Vision, Speech and Signal Processing (CVSSP), University of Surrey（视觉、语音和信号处理中心（CVSSP）、塞维利亚大学）
ETH Zurich（苏黎世联邦理工学院）；MPI for Intelligent Systems（马克斯·普朗克智能系统研究所）；University of Oxford（牛津大学）
Text classifiers have promising applications in high-stake tasks such as resume screening and content moderation. These classifiers must be fair and avoid discriminatory decisions by being invariant to perturbations of sensitive attributes such as gender or ethnicity. However, there is a gap between human intuition about these perturbations and the formal similarity specifications capturing them. While existing research has started to address this gap, current methods are based on hardcoded word replacements, resulting in specifications with limited expressivity or ones that fail to fully alig…
Toyota Technological Institute at Chicago（芝加哥丰田技术研究所）
Segments that span contiguous parts of inputs, such as phonemes in speech, named entities in sentences, actions in videos, occur frequently in sequence prediction problems. Segmental models, a class of models that explicitly hypothesizes segments, have allowed the exploration of rich segment features for sequence prediction. However, segmental models suffer from slow decoding, hampering the use of computationally expensive features. In this thesis, we introduce discriminative segmental cascades, a multi-pass inference framework that allows us to improve accuracy by adding higher-order features…
