There are two notoriously hard problems in cluster analysis, estimating the number of clusters, and checking whether the population to be clustered is not actually homogeneous. Given a dataset, a clustering method and a cluster validation index, this paper proposes to set up null models that capture structural features of the data that cannot be interpreted as indicating clustering. Artificial datasets are sampled from the null model with parameters estimated from the original dataset. This can be used for testing the null hypothesis of a homogeneous population against a clustering alternative. It can also be used to calibrate the validation index for estimating the number of clusters, by taking into account the expected distribution of the index under the null model for any given number of clusters. The approach is illustrated by three examples, involving various different clustering techniques (partitioning around medoids, hierarchical methods, a Gaussian mixture model), validation indexes (average silhouette width, prediction strength and BIC), and issues such as mixed type data, temporal and spatial autocorrelation.
Statistical models of unobserved heterogeneity are typically formalized as mixtures of simple parametric models and interest naturally focuses on testing for homogeneity versus general mixture alternatives. Many tests of this type can be interpreted as $C(α)$ tests, as in Neyman (1959), and shown to be locally, asymptotically optimal. These $C(α)$ tests will be contrasted with a new approach to likelihood ratio testing for general mixture models. The latter tests are based on estimation of general nonparametric mixing distribution with the Kiefer and Wolfowitz (1956) maximum likelihood estimat…
A method for empirically assessing small area estimators via bootstrap-weighted k-Nearest-Neighbor artificial populations, with applications to forest inventory
Assessing homogeneity of distributions is an old problem that has received considerable attention, especially in the nonparametric Bayesian literature. To this effect, we propose the semi-hierarchical Dirichlet process, a novel hierarchical prior that extends the hierarchical Dirichlet process of Teh et al. (2006) and that avoids the degeneracy issues of nested processes recently described by Camerlenghi et al. (2019a). We go beyond the simple yes/no answer to the homogeneity question and embed the proposed prior in a random partition model; this procedure allows us to give a more comprehensiv…
The Bayesian expected power (BEP) has become increasingly popular in sample size determination and assessment of the probability of success (POS) for a future trial. The BEP takes into consideration the uncertainty around the parameters assumed by a power analysis and is thus more robust compared to the traditional power that assumes a single set of parameters. Current methods for assessing BEP are often based in a parametric framework by imposing a model on the pilot data to derive and sample from the posterior distributions of the parameters. Implementation of the model-based approaches can …
The paper considers a paired data framework and discuss the question of marginal homogeneity of bivariate high dimensional or functional data. The related testing problem can be endowed into a more general setting for paired random variables taking values in a general Hilbert space. To address this problem, a Cramer-von-Mises type test statistic is applied and a bootstrap procedure is suggested to obtain critical values and finally a consistent test. The desired properties of a bootstrap test can be derived, that are asymptotic exactness under the null hypothesis and consistency under alternat…
Assessing how adequate clusters fit a dataset and finding an optimum number of clusters is a difficult process. A membership matrix and the degree of membership matrix is suggested to determine the homogeneity of a cluster fit. Maximisation of the ratio of the overall degree of membership at cluster number lag 1 is also suggested as a method to optimise the number of clusters in a dataset. A threshold factor upon the degree of membership is also suggested for homogeneous clusters. Cluster simulations were given to compare how well the proposed method compares against established methods. This …
A repeated k-fold cross-validation approach for evaluating the instability of clinical prediction models: an empirical comparison to the bootstrap approach
Faculty of Medicine, Chiang Mai University（清迈大学医学院）；Lampang Hospital（南奔医院）；Chaiprakarn Hospital（柴帕卡恩医院）
The bootstrap provides a simple and powerful means of assessing the quality of estimators. However, in settings involving large datasets---which are increasingly prevalent---the computation of bootstrap-based quantities can be prohibitively demanding computationally. While variants such as subsampling and the $m$ out of $n$ bootstrap can be used in principle to reduce the cost of bootstrap computations, we find that these methods are generally not robust to specification of hyperparameters (such as the number of subsampled data points), and they often require use of more prior information (suc…
Finite mixtures of multivariate normal distributions have been widely used in empirical applications in diverse fields such as statistical genetics and statistical finance. Testing the number of components in multivariate normal mixture models is a long-standing challenge even in the most important case of testing homogeneity. This paper develops likelihood-based tests of the null hypothesis of $M_0$ components against the alternative hypothesis of $M_0 + 1$ components for a general $M_0 \geq 1$. For heteroscedastic normal mixtures, we propose an EM test and derive the asymptotic distribution …
The tau statistic $τ$ uses geolocation and, usually, symptom onset time to assess global spatiotemporal clustering from epidemiological data. We test different factors that could affect graphical hypothesis tests of clustering or bias clustering range estimates based on the statistic, by comparison with a baseline analysis of an open access measles dataset. From re-analysing this data we find that the spatial bootstrap sampling method used to construct the confidence interval for the tau estimate and confidence interval (CI) type can bias clustering range estimates. We suggest that the bias-co…
Bootstrap is commonly used as a tool for non-parametric statistical inference to estimate meaningful parameters in Variable Selection Models. However, for massive dataset that has exponential growth rate, the computation of Bootstrap Variable Selection (BootVS) can be a crucial issue. In this paper, we propose the method of Variable Selection with Bag of Little Bootstraps (BLBVS) on General Linear Regression and extend it to Generalized Linear Model for selecting important parameters and assessing the quality of estimators' computation efficiency by analyzing results of multiple bootstrap sub-…
Fitting parametric models by optimizing frequency domain objective functions is an attractive approach of parameter estimation in time series analysis. Whittle estimators are a prominent example in this context. Under weak conditions and the (realistic) assumption that the true spectral density of the underlying process does not necessarily belong to the parametric class of spectral densities fitted, the distribution of Whittle estimators typically depends on difficult to estimate characteristics of the underlying process. This makes the implementation of asymptotic results for the constructio…
This paper is a chapter in the forthcoming Handbook of Cluster Analysis, Hennig et al. (2015). For definitions of basic clustering methods and some further methodology, other chapters of the Handbook are referred to. To read this version of the paper without the Handbook, some knowledge of cluster analysis methodology is required. The aim of this chapter is to provide a framework for all the decisions that are required when carrying out a cluster analysis in practice. A general attitude to clustering is outlined, which connects these decisions closely to the clustering aims in a given applicat…
The analysis of continuously spatially varying processes usually considers two sources of variation, namely, the large-scale variation collected by the trend of the process, and the small-scale variation. Parametric trend models on latitude and longitude are easy to fit and to interpret. However, the use of simple parametric models for characterizing spatially varying processes may lead to misspecification problems if the model is not appropriate. Recently, Meilán-Vila et al. (2019) proposed a goodness-of-fit test based on an L2-distance for assessing a parametric trend model with correlated e…
