We give an algorithm that generates a uniformly random contingency table with specified marginals, i.e. a matrix with non-negative integer values and specified row and column sums. Such algorithms are useful in statistics and combinatorics. When $Δ^4< M/5$, where $Δ$ is the maximum of the row and column sums and $M$ is the sum of all entries of the matrix, our algorithm runs in time linear in $M$ in expectation. Most previously published algorithms for this problem are approximate samplers based on Markov chain Monte Carlo, whose provable bounds on the mixing time are typically polynomials with rather large degrees.
We study a generalisation of Simpson reversal (also known as Simpson's paradox or the Yule-Simpson effect) to $2 \times 2 \times 2$ contingency tables and characterise the cases for which it can and cannot occur with two combinatorial-geometric lemmas. We also present a conjecture based on some computational experiments on the expected likelihood of such events.
We present a new lower bound on the number of contingency tables, improving upon and extending previous lower bounds by Barvinok and Gurvits. As an application, we obtain new lower bounds on the volumes of flow and transportation polytopes. Our proofs are based on recent results on Lorentzian polynomials.
The Diaconis--Gangolli random walk is an algorithm that generates an almost uniform random graph with prescribed degrees. In this paper, we study the mixing time of the Diaconis--Gangolli random walk restricted on $n\times n$ contingency tables over $\mathbb{Z}/q\mathbb{Z}$. We prove that the random walk exhibits cutoff at $\frac{n^2}{4(1- \cos{\frac{2 π}{q}})} \log n, $ when $\log q=o\left (\frac{\sqrt{\log n}}{\log \log n}\right )$.
A binary contingency table is an m x n array of binary entries with prescribed row sums r=(r_1,...,r_m) and column sums c=(c_1,...,c_n). The configuration model for uniformly sampling binary contingency tables proceeds as follows. First, label N=\sum_{i=1}^{m} r_i tokens of type 1, arrange them in m cells, and let the i-th cell contain r_i tokens. Next, label another set of tokens of type 2 containing N=\sum_{j=1}^{n}c_j elements arranged in n cells, and let the j-th cell contain c_j tokens. Finally, pair the type-1 tokens with the type-2 tokens by generating a random permutation until the tot…
We obtain sharp asymptotic estimates on the number of $n \times n$ contingency tables with two linear margins $Cn$ and $BCn$. The results imply a second order phase transition on the number of such contingency tables, with a critical value at \ts $B_{c}:=1 + \sqrt{1+1/C}$. As a consequence, for \ts $B>B_{c}$, we prove that the classical \emph{independence heuristic} leads to a large undercounting.
We present a randomized approximation algorithm for counting contingency tables, mxn non-negative integer matrices with given row sums R=(r_1, ..., r_m) and column sums C=(c_1, ..., c_n). We define smooth margins (R,C) in terms of the typical table and prove that for such margins the algorithm has quasi-polynomial N^{O(ln N)} complexity, where N=r_1+...+r_m=c_1+...+c_n. Various classes of margins are smooth, e.g., when m=O(n), n=O(m) and the ratios between the largest and the smallest row sums as well as between the largest and the smallest column sums are strictly smaller than the golden rati…
Let R=(r_1, ..., r_m) and C=(c_1, ..., c_n) be positive integer vectors such that r_1 +... + r_m=c_1 +... + c_n. We consider the set Sigma(R, C) of non-negative mxn integer matrices (contingency tables) with row sums R and column sums C as a finite probability space with the uniform measure. We prove that a random table D in Sigma(R,C) is close with high probability to a particular matrix ("typical table'') Z defined as follows. We let g(x)=(x+1) ln(x+1)-x ln x for non-negative x and let g(X)=sum_ij g(x_ij) for a non-negative matrix X=(x_ij). Then g(X) is strictly concave and attains its maxim…
Markov basis for statistical model of contingency tables gives a useful tool for performing the conditional test of the model via Markov chain Monte Carlo method. In this paper we derive explicit forms of Markov bases for change point models and block diagonal effect models, which are typical block-wise effect models of two-way contingency tables, and perform conditional tests with some real data sets.
We consider nonnegative integer matrices with specified row and column sums and upper bounds on the entries. We show that the logarithm of the number of such matrices is approximated by a concave function of the row and column sums. We give efficiently computable estimators for this function, including one suggested by a maximum-entropy random model; we show that these estimators are asymptotically exact as the dimension of the matrices goes to infinity. We finish by showing that, for kappa >= 2 and for sufficiently small row and column sums, the number of matrices with these row and column su…
In this paper we study the computation of Markov bases for contingency tables whose cell entries have an upper bound. In general a Markov basis for unbounded contingency table under a certain model differs from a Markov basis for bounded tables. Rapallo, (2007) applied Lawrence lifting to compute a Markov basis for contingency tables whose cell entries are bounded. However, in the process, one has to compute the universal Gröbner basis of the ideal associated with the design matrix for a model which is, in general, larger than any reduced Gröbner basis. Thus, this is also infeasible in small- …
Examples of small contingency tables on binary random variables with large integer programming gaps on the lower bounds of cell entries were constructed by Sullivant. We argue here that the margins for which these constructed large gaps occur are rarely encountered, thus reopening the question of whether linear programming is an effective heuristic for detecting disclosures when releasing margins of multi-way tables. The notion of ``rarely encountered'' is made precise through the language of standard pairs.
In this short note, we compute the precise asymptotics for the number of contingency tables with non-uniform margins. More precisely, for parameter $n,δ, B,C>0$, we consider the set of matrices whose first $[n^δ]$ rows and columns have sum $[BCn]$ and the rest $n$ rows and columns have sum $[Cn]$. We compute the precise asymptotics of the cardinality of this set when $B<B_c=1+\sqrt{1+1/C}$ using the maximal entropy methods developed by Barvinok and Hartigan. The only contribution of this note is a detailed expansion of the determinant of quadratic forms in asymptotic formulas.
We represent the number of mxn non-negative integer matrices (contingency tables) with prescribed row sums and column sums as the expected value of the permanent of a non-negative random matrix with exponentially distributed entries. We bound the variance of the obtained estimator, from which it follows that if the row and column sums are bounded by a constant fixed in advance, we get a polynomial time approximation scheme for counting contingency tables. We show that the complete symmetric polynomial of a fixed degree in n variables can be epsilon-approximated coefficient-wise by a sum of pow…
Given m positive integers R=(r_i), n positive integers C=(c_j) such that sum r_i = sum c_j =N, and mn non-negative weights W=(w_{ij}), we consider the total weight T=T(R, C; W) of non-negative integer matrices (contingency tables) D=(d_{ij}) with the row sums r_i, column sums c_j, and the weight of D equal to prod w_{ij}^{d_{ij}}. We present a randomized algorithm of a polynomial in N complexity which computes a number T'=T'(R,C; W) such that T' < T < alpha(R, C) T' where alpha(R,C) = min{prod r_i! r_i^{-r_i}, prod c_j! c_j^{-c_j}} N^N/N!. In many cases, ln T' provides an asymptotically accura…
The sequential importance sampling (SIS) algorithm has gained considerable popularity for its empirical success. One of its noted applications is to the binary contingency tables problem, an important problem in statistics, where the goal is to estimate the number of 0/1 matrices with prescribed row and column sums. We give a family of examples in which the SIS procedure, if run for any subexponential number of trials, will underestimate the number of tables by an exponential factor. This result holds for any of the usual design choices in the SIS algorithm, namely the ordering of the columns …
Given a non-negative mxn matrix W=(w_ij) and positive integer vectors R=(r_1, >..., r_m) and C=(c_1, ..., c_n), we consider the total weight T(R, C; W) of mxn non-negative integer matrices (contingency tables) D with the row sums r_i, the column sums c_j, and the weight of D=(d_ij) equal to product of w_ij^d_ij. In particular, if W is a 0-1 matrix, T(R, C; W) is the number of integer feasible flows in a bipartite network. We prove a version of the Brunn-Minkowski inequality relating the numbers T(R, C; W) and T(R_k, C_k; W), where (R, C) is a convex combination of (R_k, C_k) for k=1, ..., p.
For parameters $n,δ,B,$ and $C$, let $X=(X_{k\ell})$ be the random uniform contingency table whose first $\lfloor n^δ \rfloor $ rows and columns have margin $\lfloor BCn \rfloor$ and the last $n$ rows and columns have margin $\lfloor Cn \rfloor$. For every $0<δ<1$, we establish a sharp phase transition of the limiting distribution of each entry of $X$ at the critical value $B_{c}=1+\sqrt{1+1/C}$. In particular, for $1/2<δ<1$, we show that the distribution of each entry converges to a geometric distribution in total variation distance, whose mean depends sensitively on whether $B<B_{c}$ or $B>B…
We prove an asymptotic estimate for the number of mxn non-negative integer matrices (contingency tables) with prescribed row and column sums and, more generally, for the number of integer feasible flows in a network. Similarly, we estimate the volume of the polytope of mxn non-negative real matrices with prescribed row and column sums. Our estimates are solutions of convex optimization problems and hence can be computed efficiently. As a corollary, we show that if row sums R=(r_1, ..., r_m) and column sums C=(c_1, ..., c_n) with r_1 + ... + r_m =c_1 + ... +c_n =N are sufficiently far from cons…
The $k$-way discrepancy $\disc_k (\C)$ of a rectangular array $\C$ of nonnegative entries is the minimum of the maxima of the within- and between-cluster discrepancies that can be obtained by simultaneous $k$-clusterings (proper partitions) of its rows and columns. In the main theorem, irrespective of the size of $\C$, we give the following estimate for the $k$th largest non-trivial singular value of the normalized table: $s_k \le 9\disc_{k } (\C ) (k+2 -9k\ln \disc_{k } (\C ))$, provided $\disc_{k } (\C ) <1$ and $k\le \rk (\C )$. This statement is the converse of Theorem 7 of Bolla \cite{Bol…
Counting 3-way contingency tables via quiver semi-invariants
University of Missouri-Columbia（密苏里大学哥伦比亚分校）；International Center for Mathematical Sciences, Institute of Mathematics and Informatics, Bulgarian Academy of Sciences（保加利亚科学院数学与信息学研究所国际数学科学中心）
In 2005, Chen et al introduced a sequential importance sampling (SIS) procedure to analyze zero-one two-way tables with given fixed marginal sums (row and column sums) via the conditional Poisson (CP) distribution. They showed that compared with Monte Carlo Markov chain (MCMC)-based approaches, their importance sampling method is more efficient in terms of running time and also provides an easy and accurate estimate of the total number of contingency tables with fixed marginal sums. In this paper we extend their result to zero-one multi-way ($d$-way, $d \geq 2$) contingency tables under the no…
In this paper we present two independent computational proofs that the monoid derived from $5\times 5\times 3$ contingency tables is normal, completing the classification by Hibi and Ohsugi. We show that Vlach's vector disproving normality for the monoid derived from $6\times 4\times 3$ contingency tables is the unique minimal such vector up to symmetry. Finally, we compute the full Hilbert basis of the cone associated with the non-normal monoid of the semi-graphoid for $|N|=5$. The computations are based on extensions of the packages LattE-4ti2 and Normaliz.
Does a given system of linear equations with nonnegative constraints have an integer solution? This is a fundamental question in many areas. In statistics this problem arises in data security problems for contingency table data and also is closely related to non-squarefree elements of Markov bases for sampling contingency tables with given marginals. To study a family of systems with no integer solution, we focus on a commutative semigroup generated by a finite subset of $\Z^d$ and its saturation. An element in the difference of the semigroup and its saturation is called a ``hole''. We show th…
Multi-way tables with specified marginals arise in a variety of applications in statistics and operations research. We provide a comprehensive complexity classification of three fundamental computational problems on tables: existence, counting and entry-security. One major outcome of our work is that each of the following problems is intractable already for "slim" 3-tables, with constant and smallest possible number 3 of rows: (1) deciding existence of 3-tables with given consistent 2-marginals; (2) counting all 3-tables with given 2-marginals; (3) finding whether an integer value is attained …
We count mxn non-negative integer matrices (contingency tables) with prescribed row and column sums (margins). For a wide class of smooth margins we establish a computationally efficient asymptotic formula approximating the number of matrices within a relative error which approaches 0 as m and n grow.
When an interval of integers between the lower bound $l_i$ and the upper bound $u_i$ is the support of the marginal distribution $n_i|(n_{i-1}, ...,n_1)$, Chen et al, 2005 noticed that sampling from the interval at each step, for $n_i$ during a sequential importance sampling (SIS) procedure, always produces a table which satisfies the marginal constraints. However, in general, the interval may not be equal to the support of the marginal distribution. In this case, the SIS procedure may produce tables which do not satisfy the marginal constraints, leading to rejection Chen et al 2006. In this p…
When an interval of integers between the lower bound l_i and the upper bounds u_i is the support of the marginal distribution n_i|(n_{i-1}, ...,n_1), Chen et al. 2005 noticed that sampling from the interval at each step, for n_i during the sequential importance sampling (SIS) procedure, always produces a table which satisfies the marginal constraints. However, in general, the interval may not be equal to the support of the marginal distribution. In this case, the SIS procedure may produce tables which do not satisfy the marginal constraints, leading to rejection [Chen et al. 2006]. Rejecting t…
In recent years, sequential importance sampling (SIS) has been well developed for sampling contingency tables with linear constraints. In this paper, we apply SIS procedure to 2-dimensional Ising models, which give observations of 0-1 tables and include both linear and quadratic constraints. We show how to compute bounds for specific cells by solving linear programming (LP) problems over cut polytopes to reduce rejections. The computational results, which includes both simulations and real data analysis, suggest that our method performs very well for sparse tables and when the 1's are spread o…
We investigate the number of symmetric matrices of non-negative integers with zero diagonal such that each row sum is the same. Equivalently, these are zero diagonal symmetric contingency tables with uniform margins, or loop-free regular multigraphs. We determine the asymptotic value of this number as the size of the matrix tends to infinity, provided the row sum is large enough. We conjecture that our answer is valid for all row sums.
