Unlike the classical linear model, nonlinear generative models have been addressed sparsely in the literature of statistical learning. This work aims to bringing attention to these models and their secrecy potential. To this end, we invoke the replica method to derive the asymptotic normalized cross entropy in an inverse probability problem whose generative model is described by a Gaussian random field with a generic covariance function. Our derivations further demonstrate the asymptotic statistical decoupling of the Bayesian estimator and specify the decoupled setting for a given nonlinear model.
The replica solution depicts that strictly nonlinear models establish an all-or-nothing phase transition: There exists a critical load at which the optimal Bayesian inference changes from perfect to an uncorrelated learning. Based on this finding, we design a new secure coding scheme which achieves the secrecy capacity of the wiretap channel. This interesting result implies that strictly nonlinear generative models are perfectly secured without any secure coding. We justify this latter statement through the analysis of an illustrative model for perfectly secure and reliable inference.
Georg-August University of Göttingen（哥廷根乔治-奥古斯特大学）；University of Tübingen（图宾根大学）
A recent paper by Liu et al. combines the topics of adversarial training and Bayesian Neural Networks (BNN) and suggests that adversarially trained BNNs are more robust against adversarial attacks than their non-Bayesian counterparts. Here, I analyze the proposed defense and suggest that one needs to adjust the adversarial attack to incorporate the stochastic nature of a Bayesian network to perform an accurate evaluation of its robustness. Using this new type of attack I show that there appears to be no strong evidence for higher robustness of the adversarially trained BNNs.
These are written comments about the Read Paper A Bayesian criterion for singular models by M. Drton and M. Plummer, read to the Royal Statistical Society on October 5, 2016. The discussion was delivered by Judith Rousseau.
A brief comment on A variational Bayesian approach for inverse problems with skew-t error distributions (Guha et al., Journal of Computational Physics 301 (2015) 377-393) is given in this letter.
This note is commenting on Ronald Gallant's (2015) reflections on the construction of Bayesian prior distributions from moment conditions. The main conclusion is that the paper does not deliver a working principle that could justify inference based on such priors.
The recent article "A Bayesian conjugate gradient method" by Cockayne, Oates, Ipsen, and Girolami proposes an approximately Bayesian iterative procedure for the solution of a system of linear equations, based on the conjugate gradient method, that gives a sequence of Gaussian/normal estimates for the exact solution. The purpose of the probabilistic enrichment is that the covariance structure is intended to provide a posterior measure of uncertainty or confidence in the solution mean. This note gives some comments on the article, poses some questions, and suggests directions for further researc…
We discuss the methods of Evans and Moshonov [Bayesian Analysis 1 (2006) 893--914, Bayesian Statistics and Its Applications (2007) 145--159] concerning checking for prior-data conflict and their relevance to the method proposed in this paper. [arXiv:0802.0743]
We wish to contribute to the discussion of "Comparing Consensus Monte Carlo Strategies for Distributed Bayesian Computation" by offering our views on the current best methods for Bayesian computation, both at big-data scale and with smaller data sets, as summarized in Table 1. This table is certainly an over-simplification of a highly complicated area of research in constant (present and likely future) flux, but we believe that constructing summaries of this type is worthwhile despite their drawbacks, if only to facilitate further discussion.
This note discusses Watson and Holmes (2016) and their pro- posals towards more robust Bayesian decisions. While we acknowledge and commend the authors for setting new and all-encompassing prin- ciples of Bayesian robustness, and we appreciate the strong anchoring of those within a decision-theoretic referential, we remain uncertain as to which extent such principles can be applied outside binary de- cisions. We also wonder at the ultimate relevance of Kullback-Leibler neighbourhoods to characterise robustness and favour extensions along non-parametric axes.
This short note points out two of the incongruences that I find in the Loredo (2012) comments on Andreon (2012), i.e. on my chapter written for the book "Astrostatistical Challenges for the New Astronomy". First, I find illogic the Loredo decision of putting my chapter among those presenting simple models, because one of the models illustrated in my chapter is qualified by him as "impressing for his complexity". Second, Loredo criticizes my chapter at one location confusing it with another paper by another author, because my chapter do not touch the subject mentioned by Loredo (2012) critics, …
This paper is an invited commentary on Tamas Budavari's presentation, "On statistical cross-identification in astronomy," for the Statistical Challenges in Modern Astronomy V conference held at Pennsylvania State University in June 2011. I begin with a brief review of previous work on probabilistic (Bayesian) assessment of directional and spatio-temporal coincidences in astronomy (e.g., cross-matching or cross-identification of objects across multiple catalogs). Then I discuss an open issue in the recent innovative work of Budavari and his colleagues on large-scale probabilistic cross-identifi…
In this rejoinder we summarize the comments, questions and remarks on the paper "A novel algorithmic approach to Bayesian Logic Regression" from the discussants. We then respond to those comments, questions and remarks, provide several extensions of the original model and give a tutorial on our R-package EMJMCMC (http://aliaksah.github.io/EMJMCMC2016/)
Due to their great flexibility, nonparametric Bayes methods have proven to be a valuable tool for discovering complicated patterns in data. The term "nonparametric Bayes" suggests that these methods inherit model-free operating characteristics of classical nonparametric methods, as well as coherent uncertainty assessments provided by Bayesian procedures. However, as the authors say in the conclusion to their article, nonparametric Bayesian methods may be more aptly described as "massively parametric." Furthermore, I argue that many of the default nonparametric Bayes procedures are only Bayesia…
In a recent manuscript, Gelman & Yao (2020) claim that "the usual rules of conditional probability fail in the quantum realm" and that "probability theory isn't true (quantum physics)" and purport to support these statements with the example of a quantum double-slit experiment. The present comment recalls some relevant literature in quantum theory and shows that (i) Gelman & Yao's statements are false; in fact, the quantum example confirms the rules of probability theory; (ii) the particular inequality found in the quantum example can be shown to appear also in very non-quantum examples, such …
In a Perspectives article in Science, Bradley Efron concludes that Bayesian calculations cannot be uncritically accepted when using uninformative priors. We argue that this conclusion is problematic because Efron's example does not use data, hence it is not Bayesian statistics; his priors make little sense and are not uninformative; and using the available data point and an uninformative prior actually leads to a reasonable posterior distribution.
Comment on Vogels et al.'s (2024) "Bayesian Structure Learning in Undirected Gaussian Graphical Models: Literature Review with Empirical Comparison": An updated performance check using BGGM
We begin by introducing the main ideas of the paper under discussion, and we give a brief description of the method proposed. Next, we discuss an alternative approach based on B-spline expansion, and lastly we make some comments on the method's convergence rate.
We present simple conditions for Bayesian consistency in the supremum metric. The key to the technique is a triangle inequality which allows us to explicitly use weak convergence, a consequence of the standard Kullback--Leibler support condition for the prior. A further condition is to ensure that smoothed versions of densities are not too far from the original density, thus dealing with densities which could track the data too closely. A key result of the paper is that we demonstrate supremum consistency using weaker conditions compared to those currently used to secure $\mathbb{L}_1$ consist…
Datasets are growing not just in size but in complexity, creating a demand for rich models and quantification of uncertainty. Bayesian methods are an excellent fit for this demand, but scaling Bayesian inference is a challenge. In response to this challenge, there has been considerable recent work based on varying assumptions about model structure, underlying computational resources, and the importance of asymptotic correctness. As a result, there is a zoo of ideas with few clear overarching principles. In this paper, we seek to identify unifying principles, patterns, and intuitions for scalin…
We are deeply appreciative of the initiative of the editor, Marina Vanucci, in commissioning a discussion of our paper, and extremely grateful to all the discussants for their insightful and thought-provoking comments. We respond to the discussions in alphabetical order [arXiv:1409.5291].
We would like to congratulate the authors of "A Bayesian Conjugate Gradient Method" on their insightful paper, and welcome this publication which we firmly believe will become a fundamental contribution to the growing field of probabilistic numerical methods and in particular the sub-field of Bayesian numerical methods. In this short piece, which will be published as a comment alongside the main paper, we first initiate a discussion on the choice of priors for solving linear systems, then propose an extension of the Bayesian conjugate gradient (BayesCG) algorithm for solving several related li…
This comment emphasizes the importance of model checking and model fitting when making inferences about finite population quantities. It also suggests the value of using unit level models when making inferences for small subpopulations, that is, "small area" analyses [arXiv:1108.2356].
University of Western Ontario（西安大略大学）；University of Sharjah（沙迦大学）
Network attacks have been very prevalent as their rate is growing tremendously. Both organization and individuals are now concerned about their confidentiality, integrity and availability of their critical information which are often impacted by network attacks. To that end, several previous machine learning-based intrusion detection methods have been developed to secure network infrastructure from such attacks. In this paper, an effective anomaly detection framework is proposed utilizing Bayesian Optimization technique to tune the parameters of Support Vector Machine with Gaussian Kernel (SVM…
We study frequentist risk properties of predictive density estimators for mean mixtures of multivariate normal distributions, involving an unknown location parameter $θ\in \mathbb{R}^d$, and which include multivariate skew normal distributions. We provide explicit representations for Bayesian posterior and predictive densities, including the benchmark minimum risk equivariant (MRE) density, which is minimax and generalized Bayes with respect to an improper uniform density for $θ$. For four dimensions or more, we obtain Bayesian densities that improve uniformly on the MRE density under Kullback…
This note discusses the paper "Penalising model component complexity" by Simpson et al. (2017). While we acknowledge the highly novel approach to prior construction and commend the authors for setting new-encompassing principles that will Bayesian modelling, and while we perceive the potential connection with other branches of the literature, we remain uncertain as to what extent the principles exposed in the paper can be developed outside specific models, given their lack of precision. The very notions of model component, base model, overfitting prior are for instance conceptual rather than m…
Don Fraser has given an interesting account of the agreements and disagreements between Bayesian posterior probabilities and confidence levels. In this comment I discuss some cases where the lack of such agreement is extreme. I then discuss a few cases where it is possible to have Bayes procedures with frequentist validity. Such frequentist-Bayesian---or Frasian---methods deserve more attention [arXiv:1112.5582].
Item response theory (IRT) is a non-linear generative probabilistic paradigm for using exams to identify, quantify, and compare latent traits of individuals, relative to their peers, within a population of interest. In pre-existing multidimensional IRT methods, one requires a factorization of the test items. For this task, linear exploratory factor analysis is used, making IRT a posthoc model. We propose skipping the initial factor analysis by using a sparsity-promoting horseshoe prior to perform factorization directly within the IRT model so that all training occurs in a single self-consisten…
The application of Bayesian inference for the purpose of model selection is very popular nowadays. In this framework, models are compared through their marginal likelihoods, or their quotients, called Bayes factors. However, marginal likelihoods depends on the prior choice. For model selection, even diffuse priors can be actually very informative, unlike for the parameter estimation problem. Furthermore, when the prior is improper, the marginal likelihood of the corresponding model is undetermined. In this work, we discuss the issue of prior sensitivity of the marginal likelihood and its role …
Preferential sampling is a common feature in geostatistics and occurs when the locations to be sampled are chosen based on information about the phenomena under study. In this case, point pattern models are commonly used as the probability law for the distribution of the locations. However, analytic intractability of the point process likelihood prevents its direct calculation. Many Bayesian (and non-Bayesian) approaches in non-parametric model specifications handle this difficulty with approximation-based methods. These approximations involve errors that are difficult to quantify and can lead…
The flexibility of the Bayesian approach to account for covariates with measurement error is combined with semiparametric regression models for a class of continuous, discrete and mixed univariate response distributions with potentially all parameters depending on a structured additive predictor. Markov chain Monte Carlo enables a modular and numerically efficient implementation of Bayesian measurement error correction based on the imputation of unobserved error-free covariate values. We allow for very general measurement errors, including correlated replicates with heterogeneous variances. Th…
Inverse probability problems whose generative models are given by strictly nonlinear Gaussian random fields show the all-or-nothing behavior: There exists a critical rate at which Bayesian inference exhibits a phase transition. Below this rate, the optimal Bayesian estimator recovers the data perfectly, and above it the recovered data becomes uncorrelated. This study uses the replica method from the theory of spin glasses to show that this critical rate is the channel capacity. This interesting finding has a particular application to the problem of secure transmission: A strictly nonlinear Gau…
In this paper, the problem of secure sensor activation is studied for an Internet of Battlefield Things (IoBT) system in which an attacker compromises a set of the IoBT sensors for the purpose of eavesdropping and acquiring information about the battlefield. In the considered model, each IoBT sensor seeks to decide whether to transmit or not based on its utility. The utility of each sensor is expressed in terms of the redundancy of the data transmitted, the secrecy capacity and the energy consumed. Due to the limited communication range of the IoBT sensors and their uncertainty about the locat…
St. Petersburg Electrotechnical University（圣彼得堡电工大学）；ITMO University（ITMO大学）；University of Eastern Finland（东芬兰大学）；NEC Corp.（日本电气公司）
How secure automatic speaker verification (ASV) technology is? More concretely, given a specific target speaker, how likely is it to find another person who gets falsely accepted as that target? This question may be addressed empirically by studying naturally confusable pairs of speakers within a large enough corpus. To this end, one might expect to find at least some speaker pairs that are indistinguishable from each other in terms of ASV. To a certain extent, such aim is mirrored in the standardized ASV evaluation benchmarks. However, the number of speakers in such evaluation benchmarks repr…
