Enrico Carlini, Fabio Rapallo
We study the geometric structure of the statistical models for two-by-two contingency tables. One or two odds ratios are fixed and the corresponding models are shown to be a portion of a ruled quadratic surface or a segment. Some pointers to the general case of two-way contingency tables are also given and an application to case-control studies is presented.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
继续探索
In this work we study several types of diagonal-effect models for two-way contingency tables in the framework of Algebraic Statistics. We use both toric models and mixture models to encode the different behavior of the diagonal cells. We compute the invariants of these models and we explore their geometrical structure.
We consider the asymptotic distribution of a cell in a 2 x ... x 2 contingency table as the fixed marginal totals tend to infinity. The asymptotic order of the cell variance is derived and a useful diagnostic is given for determining whether the cell has a Poisson limit or a Gaussian limit. There are three forms of Poisson convergence. The exact form is shown to be determined by the growth rates of the two smallest marginal totals. The results are generalized to contingency tables with arbitrary sizes and are further complemented with concrete examples.
In contingency table analysis, the odds ratio is a commonly applied measure used to summarize the degree of association between two categorical variables, say R and S. Suppose now that for each individual in the table, a vector of continuous variables X is also observed. It is then vital to analyze whether and how the degree of association varies with X. In this work, we extend the classical odds ratio to the conditional case, and develop nonparametric estimators of this "pointwise odds ratio" to summarize the strength of local association between R and S given X. To allow for maximum flexibil…
We consider the lattice, $\mathcal{L}$, of all subsets of a multidimensional contingency table and establish the properties of monotonicity and supermodularity for the marginalization function, $n(\cdot)$, on $\mathcal{L}$. We derive from the supermodularity of $n(\cdot)$ some generalized Fréchet inequalities complementing and extending inequalities of Dobra and Fienberg. Further, we construct new monotonic and supermodular functions from $n(\cdot)$, and we remark on the connection between supermodularity and some correlation inequalities for probability distributions on lattices. We also appl…
It is an important inferential problem to test no association between two binary variables based on data. Tests based on the sample odds ratio are commonly used. We bring in a competing test based on the Pearson correlation coefficient. In particular, the Odds ratio does not extend to higher order contingency tables, whereas Pearson correlation does. It is important to understand how Pearson correlation stacks against the odds ratio in 2x2 tables. Another measure of association is the canonical correlation. In this paper, we examine how competitive Pearson correlation is vis-à-vis odds ratio i…
We present a new approach for random sampling of contingency tables of any size and constraints based on a recently introduced $\textit{probabilistic divide-and-conquer}$ technique. A simple exact sampling algorithm is presented for $2\times n$ tables, as well as a generalization where each entry of the table has a specified marginal distribution.
For statistical analysis of multiway contingency tables we propose modeling interaction terms in each maximal compact component of a hierarchical model. By this approach we can search for parsimonious models with smaller degrees of freedom than the usual hierarchical model, while preserving conditional independence structures in the hierarchical model. We discuss estimation and exacts tests of the proposed model and illustrate the advantage of the proposed modeling with some data sets.
Contingency tables are useful objects in statistics for representing 2-way data. With fixed row and column sums, and a total of $n$ entries, contingency tables correspond to parabolic double cosets of $S_n$. The uniform distribution on $S_n$ induces the Fisher-Yates distribution, classical for its use in the chi-squared test for independence. A Markov chain on $S_n$ can then induce a random process on the space of contingency tables through the double cosets correspondence. The random transpositions Markov chain on $S_n$ induces a natural `swap' Markov chain on the space of contingency tables;…
Markov basis for statistical model of contingency tables gives a useful tool for performing the conditional test of the model via Markov chain Monte Carlo method. In this paper we derive explicit forms of Markov bases for change point models and block diagonal effect models, which are typical block-wise effect models of two-way contingency tables, and perform conditional tests with some real data sets.
In this paper we study a new class of statistical models for contingency tables. We define this class of models through a subset of the binomial equations of the classical independence model. We use some notions from Algebraic Statistics to compute their sufficient statistic, and to prove that they are log-linear. Moreover, we show how to compute maximum likelihood estimates and to perform exact inference through the Diaconis-Sturmfels algorithm. Examples show that these models can be useful in a wide range of applications.
We construct examples of contingency tables on $n$ binary random variables where the gap between the linear programming lower/upper bound and the true integer lower/upper bounds on cell entries is exponentially large. These examples provide evidence that linear programming may not be an effective heuristic for detecting disclosures when releasing margins of multi-way tables.
We obtain sharp asymptotic estimates on the number of $n \times n$ contingency tables with two linear margins $Cn$ and $BCn$. The results imply a second order phase transition on the number of such contingency tables, with a critical value at \ts $B_{c}:=1 + \sqrt{1+1/C}$. As a consequence, for \ts $B>B_{c}$, we prove that the classical \emph{independence heuristic} leads to a large undercounting.
We will use the factors obtained by correspondence analysis to find biclustering of a contingency table such that the row-column cluster pairs are regular, i.e., they have small discrepancy. In our main theorem, the constant of the so-called volume-regularity is related to the SVD of the normalized contingency table. Our result is applicable to two-way cuts when both the rows and columns are divided into the same number of clusters, thus extending partly the result of Butler estimating the discrepancy of a contingency table by the second largest singular value of the normalized table (one-clus…
In this paper, it is proved that, if a toric ideal possesses a fundamental binomial none of whose monomials is squarefree, then the corresponding semigroup ring is not very ample. Moreover, very ample semigroup rings of Lawrence type are discussed. As an application, we study very ampleness of configurations arising from contingency tables.
In this paper we study the computation of Markov bases for contingency tables whose cell entries have an upper bound. In general a Markov basis for unbounded contingency table under a certain model differs from a Markov basis for bounded tables. Rapallo, (2007) applied Lawrence lifting to compute a Markov basis for contingency tables whose cell entries are bounded. However, in the process, one has to compute the universal Gröbner basis of the ideal associated with the design matrix for a model which is, in general, larger than any reduced Gröbner basis. Thus, this is also infeasible in small- …
Zero patterns in multi-way binary contingency tables with uniform margins
本文研究多路二元列联表在均匀边际下保持依赖结构的变换问题，提出新理论揭示零模式对变换存在性与统计可解释性的影响，并利用组合与线性规划技术刻画唯一解。
In this work we define log-linear models to compare several square contingency tables under the quasi-independence or the quasi-symmetry model, and the relevant Markov bases are theoretically characterized. Through Markov bases, an exact test to evaluate if two or more tables fit a common model is introduced. Two real-data examples illustrate the use of these models in different fields of applications.
In this paper we show that the agglomeration of rows or columns of a contingency table with a hierarchical clustering algorithm yields statistical models defined through toric ideals. In particular, starting from the classical independence model, the agglomeration process adds a linear part to the toric ideal generated by the $2 \times 2$ minors.
The analysis of contingency tables is a powerful statistical tool used in experiments with categorical variables. This study improves parts of the theory underlying the use of contingency tables. Specifically, the linkage disequilibrium parameter as a measure of two-way interactions applied to three-way tables makes it possible to quantify Simpsons paradox by a simple formula. With tests on three-way interactions, there is only one that determines whether the partial interactions of all variables agree or whether there is at least one variable whose partial interactions disagree. To date, ther…
We propose iterative proportional scaling (IPS) via decomposable submodels for maximizing likelihood function of a hierarchical model for contingency tables. In ordinary IPS the proportional scaling is performed by cycling through the elements of the generating class of a hierarchical model. We propose to adjust more marginals at each step. This is accomplished by expressing the generating class as a union of decomposable submodels and cycling through the decomposable models. We prove convergence of our proposed procedure, if the amount of scaling is adjusted properly at each step. We also ana…
Several interesting models for contingency tables are defined by a system of equality and inequality constraints on a suitable set of marginal log-linear parameters. After reviewing the most common difficulties which are intrinsic to order restricted testing problems, we propose two new families of testing procedures, based on similar attempts appeared in the econometric literature, in order to increase the probability of detecting several relevant violations of the supposed order relations. One set of procedures is based on the decomposition of the log-likelihood ratio when testing the given …
Quasi-symmetry and geometric marginal homogeneity: A simplicial approach to square contingency tables
本文在Aitchison几何的单纯形框架下，将拟对称性解释为e-平坦子空间，提出几何边际齐性新概念，并证明偏对称性度量可正交分解为偏离拟对称性与偏离几何边际齐性两部分。
Examples of small contingency tables on binary random variables with large integer programming gaps on the lower bounds of cell entries were constructed by Sullivant. We argue here that the margins for which these constructed large gaps occur are rarely encountered, thus reopening the question of whether linear programming is an effective heuristic for detecting disclosures when releasing margins of multi-way tables. The notion of ``rarely encountered'' is made precise through the language of standard pairs.
Statistical data is often analyzed as a contingency table, sometimes with empty cells called zeros. Such sparse tables can be due to scarse observations classified in numerous categories, as for example in genetic association studies. Thus, classical independence tests involving Pearson's chi-square statistic Q or Kullback's minimum discrimination information statistic G cannot be applied because some of the expected frequencies are too small. More generally, we consider goodness of fit tests with composite hypotheses for sparse multinomial vectors and suggest simple corrections for Q and G th…
Statistical data is often analyzed as a contingency table, sometimes with empty cells called zeros. Such sparse tables can be due to scarse observations classified in numerous categories, as for example in genetic association studies. Thus, classical independence tests involving Pearson's chi-square statistic Q or Kullback's minimum discrimination information statistic G cannot be applied because some of the expected frequencies are too small. More generally, we consider goodness of fit tests with composite hypotheses for sparse multinomial vectors and suggest simple corrections for Q and G th…
Exact tests greatly improve the analysis of contingency tables when marginals are low. For instance, researchers often use Fisher's exact test, which is conditional, or Barnard's test, which is unconditional but needs to deal with a nuisance parameter. Here, we describe the m-test, an exact, unconditional test for the study of d x m binomial contingency tables. When comparing binomial trials, the m-test is related to Barnard's exact test. However, the nuisance parameter is integrated over all its possible values, instead of maximized or otherwise estimated. According to Monte Carlo simulations…
The purpose of these notes is to present an assessment of the probability of a candidate be elected in a two-round presidential election. In the first round, all candidates can be voted on. If one of them has more than 50% of the vote (s)he is elected and there is no second round. If none of the candidates obtain more than 50% of the votes, then the top two candidates will be selected for a second round. In this second round, the most voted candidate is elected. This is the scenario of the Brazilian elections that are taking place at the moment. We are calculating the odds associated with the …
In this paper we provide a definition of pattern of outliers in contingency tables within a model-based framework. In particular, we make use of log-linear models and exact goodness-of-fit tests to specify the notions of outlier and pattern of outliers. The language and some techniques from Algebraic Statistics are essential tools to make the definition clear and easily applicable. Some numerical examples show how to use our definitions.
The sequential importance sampling (SIS) algorithm has gained considerable popularity for its empirical success. One of its noted applications is to the binary contingency tables problem, an important problem in statistics, where the goal is to estimate the number of 0/1 matrices with prescribed row and column sums. We give a family of examples in which the SIS procedure, if run for any subexponential number of trials, will underestimate the number of tables by an exponential factor. This result holds for any of the usual design choices in the SIS algorithm, namely the ordering of the columns …
A reference set, or a fiber, of a contingency table is the space of all realizations of the table under a given set of constraints such as marginal totals. Understanding the geometry of this space is a key problem in algebraic statistics, important for conducting exact conditional inference, calculating cell bounds, imputing missing cell values, and assessing the risk of disclosure of sensitive information. Motivated primarily by disclosure limitation problems where constraints can come from summary statistics other than the margins, in this paper we study the space $\mathcal{F_T}$ of all poss…
We study Markov bases of decomposable graphical models consisting of primitive moves (i.e., square-free moves of degree two) by determining the structure of fibers of sample size two. We show that the number of elements of fibers of sample size two are powers of two and we characterize primitive moves in Markov bases in terms of connected components of induced subgraphs of the independence graph of a hierarchical model. This allows us to derive a complete description of minimal Markov bases and minimal invariant Markov bases for decomposable models.
We demonstrate a novel approach for the random sampling of Latin squares of order~$n$ via probabilistic divide-and-conquer. The algorithm divides the entries of the table modulo powers of $2$, and samples a corresponding binary contingency table at each level. The sampling distribution is based on the Boltzmann sampling heuristic, along with probabilistic divide-and-conquer.
In this article, a model is proposed using Bayesian techniques to account for the high correlation between many observed set of contingency tables. In many real life data this high correlation is encountered. Simulation studies are also given to check the effectiveness of this model.
Log-linear modeling is a popular method for the analysis of contingency table data. When the table is sparse, and the data falls on a proper face $F$ of the convex support, there are consequences on model inference and model selection. Knowledge of the cells determining $F$ is crucial to mitigating these effects. We introduce the R package (R Core Team (2016)) eMLEloglin for determining $F$ and passing that information on to the glm package to fit the model properly.
Test for symmetry in $2 \times 2$ contingency tables with nonignorable nonresponses
针对含不可忽略无应答的 $2 \ imes 2$ 列联表，本文借鉴 Takai 和 Kano（2008）的识别思路，围绕三种模型提出对称性检验方法。
Obtaining $(ε,δ)$-differential privacy guarantees when using a Poisson mechanism to synthesize contingency tables
本文研究利用泊松合成机制保护列联表计数，并通过泊松分布累积分布函数获得$(ε,δ)$-概率差分隐私保证，且在行政型机密数据库上进行了实证验证。
Statistical models with latent structure have a history going back to the 1950s and have seen widespread use in the social sciences and, more recently, in computational biology and in machine learning. Here we study the basic latent class model proposed originally by the sociologist Paul F. Lazarfeld for categorical variables, and we explain its geometric structure. We draw parallels between the statistical and geometric properties of latent class models and we illustrate geometrically the causes of many problems associated with maximum likelihood estimation and related statistical inference. …
E-variables are tools for retaining type-I error guarantee with optional stopping. We extend E-variables for sequential two-sample tests to general null hypotheses and anytime-valid confidence sequences. We provide implementations for estimating risk difference, relative risk and odds-ratios in contingency tables.
Visualization for departures from symmetry with the power-divergence-type measure in two-way contingency tables
本文提出一种基于幂散度型度量的新型对应分析方法，用于在名义类别方列联表中可视化对称性偏离，其主轴贡献率相等、标度与样本量无关，并构建置信区域以提升分析准确性。
Generalized Cramér's coefficient via $f$-divergence for contingency tables
本文利用比 power-divergence 更广泛的 $f$-divergence 构造列联表关联强度度量，证明其具有良好性质，并通过 $θ$-divergence 数值实验及与潜变量相关系数的关系展示其解释优势。
Multi-way contingency tables with uniform margins
本文研究保持相依结构的均匀边际列联表转换，证明三维二分情形比二维表需要更多约束，唯一性因而依赖主观选择。
A Normal Test for Independence via Generalized Mutual Information
针对大型或稀疏列联表缺乏通用独立性检验工具的问题，本文提出基于广义互信息的渐近正态、一致性检验，并证明其收敛快于 Pearson 卡方检验。
Loglinear modelling of huge contingency tables
本文提出了一种高效推断巨大列联表高阶对数线性模型的方法，通过采样空细胞和泊松回归实现参数估计。
New Equivalence Tests for Approximate Independence in Contingency Tables
研究双向列联表中近似独立性的新等价检验，借助渐近计算临界值、自助法及边界点估计器提升性能，经模拟比较不同表大小下检验性能，并应用于实际数据集，检验在R中实现且可在线获取。
For parameters $n,δ,B,$ and $C$, let $X=(X_{k\ell})$ be the random uniform contingency table whose first $\lfloor n^δ \rfloor $ rows and columns have margin $\lfloor BCn \rfloor$ and the last $n$ rows and columns have margin $\lfloor Cn \rfloor$. For every $0<δ<1$, we establish a sharp phase transition of the limiting distribution of each entry of $X$ at the critical value $B_{c}=1+\sqrt{1+1/C}$. In particular, for $1/2<δ<1$, we show that the distribution of each entry converges to a geometric distribution in total variation distance, whose mean depends sensitively on whether $B<B_{c}$ or $B>B…
The paper considers general multiplicative models for complete and incomplete contingency tables that generalize log-linear and several other models and are entirely coordinate free. Sufficient conditions of the existence of maximum likelihood estimates under these models are given, and it is shown that the usual equivalence between multinomial and Poisson likelihoods holds if and only if an overall effect is present in the model. If such an effect is not assumed, the model becomes a curved exponential family and a related mixed parameterization is given that relies on non-homogeneous odds rat…
Bounding interventional queries from generalized incomplete contingency tables
本文针对广义不完全列联表中的随机零行问题，提出将未知概率作为自由参数并推导符号表达式，通过极值化获得干预查询和平均处理效应的锐利边界，为随机数据缺口下的因果推断提供保守且严谨的量化方法。
Generating contingency tables with fixed marginal probabilities and dependence structures described by loglinear models
本文提出利用带偏移项的对数线性Poisson回归生成指定边际概率与比值比依赖结构的列联表，并用于双系统、三系统总体规模估计量在模型假设满足和违反情形下的模拟评估。
In many applications, it is of interest to study trends over time in relationships among categorical variables, such as age group, ethnicity, religious affiliation, political party and preference for particular policies. At each time point, a sample of individuals provide responses to a set of questions, with different individuals sampled at each time. In such settings, there tends to be abundant missing data and the variables being measured may change over time. At each time point, one obtains a large sparse contingency table, with the number of cells often much larger than the number of indi…
A new technique for the detection of outliers in contingency tables is introduced. Outliers thereby are unexpected cell counts with respect to classical loglinear Poisson models. Subsets of cell counts called minimal patterns are defined, corresponding to non-singular design matrices and leading to potentially uncontaminated maximum-likelihood estimates of the model parameters and thereby the expected cell counts. A criterion to easily produce minimal patterns in the two-way case under independence is derived, based on the analysis of the positions of the chosen cells. A simulation study and a…
The model for homogeneity of proportions in a two-way contingency-table/cross-tabulation is the same as the model of independence, except that the probabilistic process generating the data is viewed as fixing the column totals (but not the row totals). When gauging the consistency of observed data with the assumption of independence, recent work has illustrated that the Euclidean/Frobenius/Hilbert-Schmidt distance is often far more statistically powerful than the classical statistics such as chi-square, the log-likelihood-ratio (G), the Freeman-Tukey/Hellinger distance, and other members of th…
A Directional Measure of Marginal Inhomogeneity for Square Contingency Tables using Discrete-time hazard
针对有序类别方形列联表，提出基于离散时间风险的方向性度量，以捕捉偏离边际同质性的大小与方向，并给出估计与置信区间。
Visualizing departures from marginal homogeneity for square contingency tables with ordered categories
针对方形列联表边际同质性分析缺乏直观可视化方法的问题，提出基于Matusita距离的度量及分类数据子度量的可视化技术，并通过示例验证其有效性与解释力。
Visualization of Extremely Sparse Contingency Table by Taxicab Correspondence Analysis: A Case Study of Textual Data
本文介绍出租车对应分析这一稳健方法，用于可视化590×8265的极稀疏神圣文本数据集，并与多种降维方法进行比较研究。
For testing independence it is very popular to use either the $χ^{2}$-statistic or $G^{2}$-statistics (mutual information). Asymptotically both are $χ^{2}$-distributed so an obvious question is which of the two statistics that has a distribution that is closest to the $χ^{2}$-distribution. Surprisingly the distribution of mutual information is much better approximated by a $χ^{2}$-distribution than the $χ^{2}$-statistic. For technical reasons we shall focus on the simplest case with one degree of freedom. We introduce the signed log-likelihood and demonstrate that its distribution function can…
Geometric Mean Type of Proportional Reduction in Variation Measure for Two-Way Contingency Tables
本文提出几何平均型PRV（geoPRV）度量，通过几何平均灵敏捕捉二维列联表中每个解释变量与响应变量的关联，避免单元格部分偏倚时的低估，并可用满足特定条件的任意函数构造，统一表达传统PRV度量。
Hypothesis testing in contingency tables is usually based on asymptotic results, thereby restricting its proper use to large samples. To study these tests in small samples, we consider the likelihood ratio test and define an accurate index, the P-value, for the celebrated hypotheses of homogeneity, independence, and Hardy-Weinberg equilibrium. The aim is to understand the use of the asymptotic results of the frequentist Likelihood Ratio Test and the Bayesian FBST -- Full Bayesian Significance Test -- under small-sample scenarios. The proposed exact P-value is used as a benchmark to understand …
In two-way contingency tables we sometimes find that frequencies along the diagonal cells are relatively larger(or smaller) compared to off-diagonal cells, particularly in square tables with the common categories for the rows and the columns. In this case the quasi-independence model with an additional parameter for each of the diagonal cells is usually fitted to the data. A simpler model than the quasi-independence model is to assume a common additional parameter for all the diagonal cells. We consider testing the goodness of fit of the common diagonal effect by Markov chain Monte Carlo (MCMC…
Random sampling of contingency tables and partitions: Two practical examples of the Burnside process
本文利用 Burnside 过程及 lumping 技术构造列联表与整数分拆的高效近似均匀抽样算法，并提出反射 Burnside 过程以显著加速收敛，可抽样大小达 $10^{10}$ 的均匀分拆。
Mutual information and the encoding of contingency tables
本文提出一种改进的列联表编码方法，能显著优化信息成本上界，使简化互信息在标签高度相似时逼近理想值，从而更准确度量分类与社区检测中的标签相似度。
扫码添加微信好友，提出您的宝贵建议 👇
💡 备注请填写：网站反馈
发表机构由 AI 从论文文本抽取，如发现错误，欢迎反馈修正建议。