We give a survey of the foundations of statistical queries and their many applications to other areas. We introduce the model, give the main definitions, and we explore the fundamental theory statistical queries and how how it connects to various notions of learnability. We also give a detailed summary of some of the applications of statistical queries to other areas, including to optimization, to evolvability, and to differential privacy.
Carnegie Mellon University（卡内基梅隆大学）；Princeton University（普林斯顿大学）；Microsoft Research New England（微软研究院新英格兰分部）；Harvard University（哈佛大学）
Suppose we would like to know all answers to a set of statistical queries C on a data set up to small error, but we can only access the data itself using statistical queries. A trivial solution is to exhaustively ask all queries in C. Can we do any better? + We show that the number of statistical queries necessary and sufficient for this task is---up to polynomial factors---equal to the agnostic learning complexity of C in Kearns' statistical query (SQ) model. This gives a complete answer to the question when running time is not a concern. + We then show that the problem can be solved efficien…
We show an essentially tight bound on the number of adaptively chosen statistical queries that a computationally efficient algorithm can answer accurately given $n$ samples from an unknown distribution. A statistical query asks for the expectation of a predicate over the underlying distribution, and an answer to a statistical query is accurate if it is "close" to the correct expectation over the distribution. This question was recently studied by Dwork et al., who showed how to answer $\tildeΩ(n^2)$ queries efficiently, and also by Hardt and Ullman, who showed that answering $\tilde{O}(n^3)$ q…
A new line of work, started with Dwork et al., studies the task of answering statistical queries using a sample and relates the problem to the concept of differential privacy. By the Hoeffding bound, a sample of size $O(\log k/α^2)$ suffices to answer $k$ non-adaptive queries within error $α$, where the answers are computed by evaluating the statistical queries on the sample. This argument fails when the queries are chosen adaptively (and can hence depend on the sample). Dwork et al. showed that if the answers are computed with $(ε,δ)$-differential privacy then $O(ε)$ accuracy is guaranteed wi…
Carnegie Mellon University（卡内基梅隆大学）；University of Pennsylvania（宾夕法尼亚大学）
We revisit the problem of accurately answering large classes of statistical queries while preserving differential privacy. Previous approaches to this problem have either been very general but have not had run-time polynomial in the size of the database, have applied only to very limited classes of queries, or have relaxed the notion of worst-case error guarantees. In this paper we consider the large class of sparse queries, which take non-zero values on only polynomially many universe elements. We give efficient query release algorithms for this class, in both the interactive and the non-inte…
Harvard University John A. Paulson School of Engineering and Applied Sciences（哈佛大学约翰·A·保尔森工程与应用科学学院）；Yale University（耶鲁大学）；Northeastern University College of Computer and Information Science（东北大学计算机与信息科学学院）
We consider the problem of answering queries about a sensitive dataset subject to differential privacy. The queries may be chosen adversarially from a larger set Q of allowable queries in one of three ways, which we list in order from easiest to hardest to answer: Offline: The queries are chosen all at once and the differentially private mechanism answers the queries in a single batch. Online: The queries are chosen all at once, but the mechanism only receives the queries in a streaming fashion and must answer each query before seeing the next query. Adaptive: The queries are chosen one at a t…
Toyota Technological Institute at Chicago（芝加哥丰田理工学院）；Google Brain（谷歌大脑）；School of Mathematical Sciences, Tel Aviv University（特拉维夫大学数学科学学院）
The problem of handling adaptivity in data analysis, intentional or not, permeates a variety of fields, including test-set overfitting in ML challenges and the accumulation of invalid scientific discoveries. We propose a mechanism for answering an arbitrarily long sequence of potentially adaptive statistical queries, by charging a price for each query and using the proceeds to collect additional samples. Crucially, we guarantee statistical validity without any assumptions on how the queries are generated. We also ensure with high probability that the cost for $M$ non-adaptive queries is $O(\lo…
IBM Research Almaden（IBM阿尔马登研究院）；Harvard University School of Engineering and Applied Sciences（哈佛大学工程与应用科学学院）；Center for Research on Computation and Society（计算与社会研究中心）
We show that, under a standard hardness assumption, there is no computationally efficient algorithm that given $n$ samples from an unknown distribution can give valid answers to $n^{3+o(1)}$ adaptively chosen statistical queries. A statistical query asks for the expectation of a predicate over the underlying distribution, and an answer to a statistical query is valid if it is "close" to the correct expectation over the distribution. Our result stands in stark contrast to the well known fact that exponentially many statistical queries can be answered validly and efficiently if the queries are c…
We give algorithms for estimating the expectation of a given real-valued function $ϕ:X\to {\bf R}$ on a sample drawn randomly from some unknown distribution $D$ over domain $X$, namely ${\bf E}_{{\bf x}\sim D}[ϕ({\bf x})]$. Our algorithms work in two well-studied models of restricted access to data samples. The first one is the statistical query (SQ) model in which an algorithm has access to an SQ oracle for the input distribution $D$ over $X$ instead of i.i.d. samples from $D$. Given a query function $ϕ:X \to [0,1]$, the oracle returns an estimate of ${\bf E}_{{\bf x}\sim D}[ϕ({\bf x})]$ with…
Topological statistical theory provides the foundation for a modern mathematical reformulation of classical statistical theory: Structural Statistics emphasizes the structural assumptions that accompany distribution families and the set of structure preserving transformations between them, given by their statistical morphisms. The resulting language is designed to integrate complicated structured model spaces like deep-learning models and to close the gap to topology and differential geometry. To preserve the compatibility to classical statistics the language comprises corresponding concepts f…
A data sketch algorithm scans a big data set, collecting a small amount of data -- the sketch, which can be used to statistically infer properties of the big data set. Some data sketch algorithms take a fixed-size random sample of a big data set, and use that sample to infer frequencies of items that meet various criteria in the big data set. This paper shows how to statistically infer probably approximately correct (PAC) bounds for those frequencies, efficiently, and precisely enough that the frequency bounds are either sharp or off by only one, which is the best possible result without exact…
Can Tabular Foundation Models Amortize Statistical Inference?
London School of Economics and Political Science（伦敦政治经济学院）；University of Science and Technology of China（中国科学技术大学）；Shanghai University of Finance and Economics（上海财经大学）；University of Milano-Bicocca（米兰比可卡大学）
University of Pennsylvania（宾夕法尼亚大学）；New Jersey Institute of Technology（新泽西理工学院）；Washington University in St. Louis（圣路易斯华盛顿大学）；Argonne National Laboratory（阿贡国家实验室）
Several well-studied models of access to data samples, including statistical queries, local differential privacy and low-communication algorithms rely on queries that provide information about a function of a single sample. (For example, a statistical query (SQ) gives an estimate of $Ex_{x \sim D}[q(x)]$ for any choice of the query function $q$ mapping $X$ to the reals, where $D$ is an unknown data distribution over $X$.) Yet some data analysis algorithms rely on properties of functions that depend on multiple samples. Such algorithms would be naturally implemented using $k$-wise queries each …
We describe a slightly sub-exponential time algorithm for learning parity functions in the presence of random classification noise. This results in a polynomial-time algorithm for the case of parity functions that depend on only the first O(log n log log n) bits of input. This is the first known instance of an efficient noise-tolerant algorithm for a concept class that is provably not learnable in the Statistical Query model of Kearns. Thus, we demonstrate that the set of problems learnable in the statistical query model is a strict subset of those problems learnable in the presence of noise i…
Neural Architectures for Amortized Bayesian Inference: Statistical Foundations and Empirical Assessments
Department of Mathematics and Statistics, Indian Institute of Technology Kanpur（数学与统计学系，印度理工学院坎浦尔分校）；Marshall School of Business, University of Southern California（马歇尔商学院，南加州大学）
We consider the well-studied problem of learning intersections of halfspaces under the Gaussian distribution in the challenging \emph{agnostic learning} model. Recent work of Diakonikolas et al. (2021) shows that any Statistical Query (SQ) algorithm for agnostically learning the class of intersections of $k$ halfspaces over $\mathbb{R}^n$ to constant excess error either must make queries of tolerance at most $n^{-\tildeΩ(\sqrt{\log k})}$ or must make $2^{n^{Ω(1)}}$ queries. We strengthen this result by improving the tolerance requirement to $n^{-\tildeΩ(\log k)}$. This lower bound is essential…
SIGMA-PPG: Statistical-prior Informed Generative Masking Architecture for PPG Foundation Model
Department of Electronics, Information and Bioengineering, Politecnico di Milano（电子工程与信息生物工程系，米兰理工大学）；State Key Laboratory of Industrial Control Technology, Zhejiang University（工业控制技术国家重点实验室，浙江大学）；Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences（深圳先进技术研究院，中国科学院）；Nell Hodgson Woodruff School of Nursing, Emory University（埃默里大学护理学院）
Tel Aviv University（特拉维夫大学）；Google Research（谷歌研究院）
The average distance from a node to all other nodes in a graph, or from a query point in a metric space to a set of points, is a fundamental quantity in data analysis. The inverse of the average distance, known as the (classic) closeness centrality of a node, is a popular importance measure in the study of social networks. We develop novel structural insights on the sparsifiability of the distance relation via weighted sampling. Based on that, we present highly practical algorithms with strong statistical guarantees for fundamental problems. We show that the average distance (and hence the cen…
