For two or more classes (or types) of points, nearest neighbor contingency tables (NNCTs) are constructed using nearest neighbor (NN) frequencies and are used in testing spatial segregation of the classes. Pielou's test of independence, Dixon's cell-specific, class-specific, and overall tests are the tests based on NNCTs (i.e., they are NNCT-tests). These tests are designed and intended for use under the null pattern of random labeling (RL) of completely mapped data. However, it has been shown that Pielou's test is not appropriate for testing segregation against the RL pattern while Dixon's tests are. In this article, we compare Pielou's and Dixon's NNCT-tests; introduce the one-sided versions of Pielou's test; extend the use of NNCT-tests for testing complete spatial randomness (CSR) of points from two or more classes (which is called \emph{CSR independence}, henceforth). We assess the finite sample performance of the tests by an extensive Monte Carlo simulation study and demonstrate that Dixon's tests are also appropriate for testing CSR independence; but Pielou's test and the corresponding one-sided versions are liberal for testing CSR independence or RL. Furthermore, we show that Pielou's tests are only appropriate when the NNCT is based on a random sample of (base, NN) pairs. We also prove the consistency of the tests under their appropriate null hypotheses. Moreover, we investigate the edge (or boundary) effects on the NNCT-tests and compare the buffer zone and toroidal edge correction methods for these tests. We illustrate the tests on a real life and an artificial data set.
The spatial clustering of points from two or more classes (or species) has important implications in many fields and may cause the spatial patterns of segregation and association, which are two major types of spatial interaction between the classes. The null patterns we consider are random labeling (RL) and complete spatial randomness (CSR) of points from two or more classes, which is called CSR independence. The segregation and association patterns can be studied using a nearest neighbor contingency table (NNCT) which is constructed using the frequencies of nearest neighbor (NN) types in a co…
The spatial interaction between two or more classes (or species) has important consequences in many fields and might cause multivariate clustering patterns such as segregation or association. The spatial pattern of segregation occurs when members of a class tend to be found near members of the same class (i.e., conspecifics), while association occurs when members of a class tend to be found near members of the other class or classes. These patterns can be tested using a nearest neighbor contingency table (NNCT). The null hypothesis is randomness in the nearest neighbor (NN) structure, which ma…
In this article, a model is proposed using Bayesian techniques to account for the high correlation between many observed set of contingency tables. In many real life data this high correlation is encountered. Simulation studies are also given to check the effectiveness of this model.
Multivariate interaction between two or more classes (or species) has important consequences in many fields and causes multivariate clustering patterns such as segregation or association. The spatial segregation occurs when members of a class tend to be found near members of the same class (i.e., near conspecifics) while spatial association occurs when members of a class tend to be found near members of the other class or classes. These patterns can be studied using a nearest neighbor contingency table (NNCT). The null hypothesis is randomness in the nearest neighbor (NN) structure, which may …
Spatial interaction patterns such as segregation and association can be tested using nearest neighbor contingency tables (NNCTs). We introduce new cell-specific (or pairwise) and overall segregation tests and determine their asymptotic distributions. In particular, we demonstrate that cell-specific tests enjoy asymptotic normality, while overall tests have chi-square distributions asymptotically. We also perform an extensive Monte Carlo simulation study to compare the finite sample performance of the tests in terms of empirical size and power. In addition to the cell-specific tests as post-hoc…
Nearest neighbor (NN) methods are employed for drawing inferences about spatial patterns of points from two or more classes. We consider Pielou's test of niche specificity which is defined using a contingency table based on the NN relationships between the data points. We demonstrate that Pielou's contingency table for niche specificity is actually more appropriate for testing reflexivity in NN structure, hence we call this table as NN reflexivity contingency table (NN-RCT) henceforth. We also derive an asymptotic approximation for the distribution of the entries of the NN-RCT and consider var…
E-variables are tools for retaining type-I error guarantee with optional stopping. We extend E-variables for sequential two-sample tests to general null hypotheses and anytime-valid confidence sequences. We provide implementations for estimating risk difference, relative risk and odds-ratios in contingency tables.
Spatial interaction between two or more classes or species has important implications in various fields and causes multivariate patterns such as segregation or association. Segregation occurs when members of a class or species are more likely to be found near members of the same class or conspecifics; while association occurs when members of a class or species are more likely to be found near members of another class or species. The null patterns considered are random labeling (RL) and complete spatial randomness (CSR) of points from two or more classes, which is called \emph{CSR independence}…
The paper considers general multiplicative models for complete and incomplete contingency tables that generalize log-linear and several other models and are entirely coordinate free. Sufficient conditions of the existence of maximum likelihood estimates under these models are given, and it is shown that the usual equivalence between multinomial and Poisson likelihoods holds if and only if an overall effect is present in the model. If such an effect is not assumed, the model becomes a curved exponential family and a related mixed parameterization is given that relies on non-homogeneous odds rat…
In this work we study several types of diagonal-effect models for two-way contingency tables in the framework of Algebraic Statistics. We use both toric models and mixture models to encode the different behavior of the diagonal cells. We compute the invariants of these models and we explore their geometrical structure.
There are two popular general approaches for the analysis and visualization of a contingency table and a compositional data set: Correspondence analysis (CA) and log ratio analysis (LRA). LRA includes two independently well developed methods: association models and compositional data analysis. The application of either CA or LRA to a contingency table or to compositional data set includes a preprocessing centering step. In CA the centering step is multiplicative, while in LRA it is log bi-additive. A preprocessed matrix is double-centered, so it is a residuel matrix; which implies that it affe…
We will use the factors obtained by correspondence analysis to find biclustering of a contingency table such that the row-column cluster pairs are regular, i.e., they have small discrepancy. In our main theorem, the constant of the so-called volume-regularity is related to the SVD of the normalized contingency table. Our result is applicable to two-way cuts when both the rows and columns are divided into the same number of clusters, thus extending partly the result of Butler estimating the discrepancy of a contingency table by the second largest singular value of the normalized table (one-clus…
In this work we define log-linear models to compare several square contingency tables under the quasi-independence or the quasi-symmetry model, and the relevant Markov bases are theoretically characterized. Through Markov bases, an exact test to evaluate if two or more tables fit a common model is introduced. Two real-data examples illustrate the use of these models in different fields of applications.
In this paper we show that the agglomeration of rows or columns of a contingency table with a hierarchical clustering algorithm yields statistical models defined through toric ideals. In particular, starting from the classical independence model, the agglomeration process adds a linear part to the toric ideal generated by the $2 \times 2$ minors.
In many applications, it is of interest to study trends over time in relationships among categorical variables, such as age group, ethnicity, religious affiliation, political party and preference for particular policies. At each time point, a sample of individuals provide responses to a set of questions, with different individuals sampled at each time. In such settings, there tends to be abundant missing data and the variables being measured may change over time. At each time point, one obtains a large sparse contingency table, with the number of cells often much larger than the number of indi…
The analysis of contingency tables is a powerful statistical tool used in experiments with categorical variables. This study improves parts of the theory underlying the use of contingency tables. Specifically, the linkage disequilibrium parameter as a measure of two-way interactions applied to three-way tables makes it possible to quantify Simpsons paradox by a simple formula. With tests on three-way interactions, there is only one that determines whether the partial interactions of all variables agree or whether there is at least one variable whose partial interactions disagree. To date, ther…
In this paper we provide a definition of pattern of outliers in contingency tables within a model-based framework. In particular, we make use of log-linear models and exact goodness-of-fit tests to specify the notions of outlier and pattern of outliers. The language and some techniques from Algebraic Statistics are essential tools to make the definition clear and easily applicable. Some numerical examples show how to use our definitions.
We present a new approach for random sampling of contingency tables of any size and constraints based on a recently introduced $\textit{probabilistic divide-and-conquer}$ technique. A simple exact sampling algorithm is presented for $2\times n$ tables, as well as a generalization where each entry of the table has a specified marginal distribution.
We construct examples of contingency tables on $n$ binary random variables where the gap between the linear programming lower/upper bound and the true integer lower/upper bounds on cell entries is exponentially large. These examples provide evidence that linear programming may not be an effective heuristic for detecting disclosures when releasing margins of multi-way tables.
Exact tests greatly improve the analysis of contingency tables when marginals are low. For instance, researchers often use Fisher's exact test, which is conditional, or Barnard's test, which is unconditional but needs to deal with a nuisance parameter. Here, we describe the m-test, an exact, unconditional test for the study of d x m binomial contingency tables. When comparing binomial trials, the m-test is related to Barnard's exact test. However, the nuisance parameter is integrated over all its possible values, instead of maximized or otherwise estimated. According to Monte Carlo simulations…
It is an important inferential problem to test no association between two binary variables based on data. Tests based on the sample odds ratio are commonly used. We bring in a competing test based on the Pearson correlation coefficient. In particular, the Odds ratio does not extend to higher order contingency tables, whereas Pearson correlation does. It is important to understand how Pearson correlation stacks against the odds ratio in 2x2 tables. Another measure of association is the canonical correlation. In this paper, we examine how competitive Pearson correlation is vis-à-vis odds ratio i…
Hypothesis testing in contingency tables is usually based on asymptotic results, thereby restricting its proper use to large samples. To study these tests in small samples, we consider the likelihood ratio test and define an accurate index, the P-value, for the celebrated hypotheses of homogeneity, independence, and Hardy-Weinberg equilibrium. The aim is to understand the use of the asymptotic results of the frequentist Likelihood Ratio Test and the Bayesian FBST -- Full Bayesian Significance Test -- under small-sample scenarios. The proposed exact P-value is used as a benchmark to understand …
We obtain sharp asymptotic estimates on the number of $n \times n$ contingency tables with two linear margins $Cn$ and $BCn$. The results imply a second order phase transition on the number of such contingency tables, with a critical value at \ts $B_{c}:=1 + \sqrt{1+1/C}$. As a consequence, for \ts $B>B_{c}$, we prove that the classical \emph{independence heuristic} leads to a large undercounting.
In two-way contingency tables we sometimes find that frequencies along the diagonal cells are relatively larger(or smaller) compared to off-diagonal cells, particularly in square tables with the common categories for the rows and the columns. In this case the quasi-independence model with an additional parameter for each of the diagonal cells is usually fitted to the data. A simpler model than the quasi-independence model is to assume a common additional parameter for all the diagonal cells. We consider testing the goodness of fit of the common diagonal effect by Markov chain Monte Carlo (MCMC…
Contingency tables are useful objects in statistics for representing 2-way data. With fixed row and column sums, and a total of $n$ entries, contingency tables correspond to parabolic double cosets of $S_n$. The uniform distribution on $S_n$ induces the Fisher-Yates distribution, classical for its use in the chi-squared test for independence. A Markov chain on $S_n$ can then induce a random process on the space of contingency tables through the double cosets correspondence. The random transpositions Markov chain on $S_n$ induces a natural `swap' Markov chain on the space of contingency tables;…
We study the geometric structure of the statistical models for two-by-two contingency tables. One or two odds ratios are fixed and the corresponding models are shown to be a portion of a ruled quadratic surface or a segment. Some pointers to the general case of two-way contingency tables are also given and an application to case-control studies is presented.
We assessed several agreement coefficients applied in 2x2 contingency tables, which are commonly applied in research due to dicotomization by the conditions of the subjects (e.g., male or female) or by conveniency of the classification (e.g., traditional thresholds leading to separations in healthy or diseased, exposed or non-exposed, etc.). More extreme table configurations (e.g., high agreement between raters) are also usual, but some of the coefficients have problems with imbalanced tables. Here, we not only studied some especific estimators, but also developed a general method to the study…
Markov basis for statistical model of contingency tables gives a useful tool for performing the conditional test of the model via Markov chain Monte Carlo method. In this paper we derive explicit forms of Markov bases for change point models and block diagonal effect models, which are typical block-wise effect models of two-way contingency tables, and perform conditional tests with some real data sets.
We demonstrate a novel approach for the random sampling of Latin squares of order~$n$ via probabilistic divide-and-conquer. The algorithm divides the entries of the table modulo powers of $2$, and samples a corresponding binary contingency table at each level. The sampling distribution is based on the Boltzmann sampling heuristic, along with probabilistic divide-and-conquer.
A new technique for the detection of outliers in contingency tables is introduced. Outliers thereby are unexpected cell counts with respect to classical loglinear Poisson models. Subsets of cell counts called minimal patterns are defined, corresponding to non-singular design matrices and leading to potentially uncontaminated maximum-likelihood estimates of the model parameters and thereby the expected cell counts. A criterion to easily produce minimal patterns in the two-way case under independence is derived, based on the analysis of the positions of the chosen cells. A simulation study and a…
Large contingency tables arise in many contexts but especially in the collection of survey and census data by government statistical agencies. Because the vast majority of the variables in this context have a large number of categories, agencies and users need a systematic way of constructing tables which are summaries of such contingency tables. We propose such an approach in this paper by finding members of a class of restricted log-linear models which maximize the likelihood of the data and use this to find a parsimonious means of representing the table. In contrast with more standard appro…
We consider the asymptotic distribution of a cell in a 2 x ... x 2 contingency table as the fixed marginal totals tend to infinity. The asymptotic order of the cell variance is derived and a useful diagnostic is given for determining whether the cell has a Poisson limit or a Gaussian limit. There are three forms of Poisson convergence. The exact form is shown to be determined by the growth rates of the two smallest marginal totals. The results are generalized to contingency tables with arbitrary sizes and are further complemented with concrete examples.
The model for homogeneity of proportions in a two-way contingency-table/cross-tabulation is the same as the model of independence, except that the probabilistic process generating the data is viewed as fixing the column totals (but not the row totals). When gauging the consistency of observed data with the assumption of independence, recent work has illustrated that the Euclidean/Frobenius/Hilbert-Schmidt distance is often far more statistically powerful than the classical statistics such as chi-square, the log-likelihood-ratio (G), the Freeman-Tukey/Hellinger distance, and other members of th…
In this paper we study a new class of statistical models for contingency tables. We define this class of models through a subset of the binomial equations of the classical independence model. We use some notions from Algebraic Statistics to compute their sufficient statistic, and to prove that they are log-linear. Moreover, we show how to compute maximum likelihood estimates and to perform exact inference through the Diaconis-Sturmfels algorithm. Examples show that these models can be useful in a wide range of applications.
Visualization and interpretation of contingency tables by correspondence analysis (CA), as developed by Benzecri, has a rich structure based on Euclidean geometry. However, it is a well established fact that, often CA is very sensitive to sparse contingency tables, where we caracterize sparsity as the existence of relatively high-valued counts, rare observations discussed by Rao (1995), and zero-block structure emphasized by Novak and Bar-Hen (2005) and Greenacre (2013). In this paper, we aim to emphasize the important roles played by L1 and L2 geometries. This will be done by comparing the ma…
We consider the lattice, $\mathcal{L}$, of all subsets of a multidimensional contingency table and establish the properties of monotonicity and supermodularity for the marginalization function, $n(\cdot)$, on $\mathcal{L}$. We derive from the supermodularity of $n(\cdot)$ some generalized Fréchet inequalities complementing and extending inequalities of Dobra and Fienberg. Further, we construct new monotonic and supermodular functions from $n(\cdot)$, and we remark on the connection between supermodularity and some correlation inequalities for probability distributions on lattices. We also appl…
Examples of small contingency tables on binary random variables with large integer programming gaps on the lower bounds of cell entries were constructed by Sullivant. We argue here that the margins for which these constructed large gaps occur are rarely encountered, thus reopening the question of whether linear programming is an effective heuristic for detecting disclosures when releasing margins of multi-way tables. The notion of ``rarely encountered'' is made precise through the language of standard pairs.
In this paper, it is proved that, if a toric ideal possesses a fundamental binomial none of whose monomials is squarefree, then the corresponding semigroup ring is not very ample. Moreover, very ample semigroup rings of Lawrence type are discussed. As an application, we study very ampleness of configurations arising from contingency tables.
In this paper we study the computation of Markov bases for contingency tables whose cell entries have an upper bound. In general a Markov basis for unbounded contingency table under a certain model differs from a Markov basis for bounded tables. Rapallo, (2007) applied Lawrence lifting to compute a Markov basis for contingency tables whose cell entries are bounded. However, in the process, one has to compute the universal Gröbner basis of the ideal associated with the design matrix for a model which is, in general, larger than any reduced Gröbner basis. Thus, this is also infeasible in small- …
Log-linear modeling is a popular method for the analysis of contingency table data. When the table is sparse, and the data falls on a proper face $F$ of the convex support, there are consequences on model inference and model selection. Knowledge of the cells determining $F$ is crucial to mitigating these effects. We introduce the R package (R Core Team (2016)) eMLEloglin for determining $F$ and passing that information on to the glm package to fit the model properly.
Methods for selecting loglinear models were among Steve Fienberg's research interests since the start of his long and fruitful career. After we dwell upon the string of papers focusing on loglinear models that can be partly attributed to Steve's contributions and influential ideas, we develop a new algorithm for selecting graphical loglinear models that is suitable for analyzing hyper-sparse contingency tables. We show how multi-way contingency tables can be used to represent patterns of human mobility. We analyze a dataset of geolocated tweets from South Africa that comprises 46 million latit…
For statistical analysis of multiway contingency tables we propose modeling interaction terms in each maximal compact component of a hierarchical model. By this approach we can search for parsimonious models with smaller degrees of freedom than the usual hierarchical model, while preserving conditional independence structures in the hierarchical model. We discuss estimation and exacts tests of the proposed model and illustrate the advantage of the proposed modeling with some data sets.
