Community structure analysis is a powerful tool for complex networks, which can simplify their functional analysis considerably. Recently, many approaches were proposed to community structure detection, but few works were focused on the significance of community structure. Since real networks obtained from complex systems always contain error links, and most of the community detection algorithms have random factors, evaluate the significance of community structure is important and urgent. In this paper, we use the eigenvectors' stability to characterize the significance of community structures. By employing the eigenvalues of Laplacian matrix of a given network, we can evaluate the significance of its community structure and obtain the optimal number of communities, which are always hard for community detection algorithms. We apply our method to many real networks. We find that significant community structures exist in many social networks and C.elegans neural network, and that less significant community structures appear in protein-interaction networks and metabolic networks. Our method can be applied to broad clustering problems in data mining due to its solid mathematical basis and efficiency.
Community structure is an important structural property that extensively exists in various complex networks. In the past decade, much attention has been paid to the design of community-detection methods, but analyzing the behaviors of the methods is also of interest in the theoretical research and real applications. Here, we focus on an important measure for community structure, significance [Sci. Rep. 3 (2013) 2930]. Specifically, we study the effect of various network parameters on this measure in detail, analyze the critical behaviors of it in partition transition, and analytically give the…
Community structure analysis is a powerful tool for social networks, which can simplify their topological and functional analysis considerably. However, since community detection methods have random factors and real social networks obtained from complex systems always contain error edges, evaluating the significance of community structure partitioned is an urgent and important question. In this paper, integrating the specific characteristics of real society, we present a novel framework analyzing the significance of social community specially. The dynamics of social interactions are modeled by…
Many complex systems can be represented as networks and separating a network into communities could simplify the functional analysis considerably. Recently, many approaches have been proposed for finding communities, but none of them can evaluate the communities found are significant or trivial definitely. In this paper, we propose an index to evaluate the significance of communities in networks. The index is based on comparing the similarity between the original community structure in network and the community structure of the network after perturbed, and is defined by integrating all the sim…
Many empirical networks have community structure, in which nodes are densely interconnected within each community (i.e., a group of nodes) and sparsely across different communities. Like other local and meso-scale structure of networks, communities are generally heterogeneous in various aspects such as the size, density of edges, connectivity to other communities and significance. In the present study, we propose a method to statistically test the significance of individual communities in a given network. Compared to the previous methods, the present algorithm is unique in that it accepts diff…
The discovery of community structure is a common challenge in the analysis of network data. Many methods have been proposed for finding community structure, but few have been proposed for determining whether the structure found is statistically significant or whether, conversely, it could have arisen purely as a result of chance. In this paper we show that the significance of community structure can be effectively quantified by measuring its robustness to small perturbations in network structure. We propose a suitable method for perturbing networks and a measure of the resulting change in comm…
It is often of interest to find communities in network data for unsupervised learning, feature discovery, anomaly detection, or scientific study. The vast majority of community detection methods proceed via optimization of a quality function, which is possible even on random networks without communities. Therefore there is usually not an easy way to tell if a community is "significant", in this context meaning more internally connected than would be expected under a random graph model without communities. This paper generalizes existing null models for this purpose to bipartite graphs, and int…
Many complex networks show signs of modular structure, uncovered by community detection. Although many methods succeed in revealing various partitions, it remains difficult to detect at what scale some partition is significant. This problem shows foremost in multi-resolution methods. We here introduce an efficient method for scanning for resolutions in one such method. Additionally, we introduce the notion of "significance" of a partition, based on subgraph probabilities. Significance is independent of the exact method used, so could also be applied in other methods, and can be interpreted as …
Community detection is the process of grouping strongly connected nodes in a network. Many community detection methods for un-weighted networks have a theoretical basis in a null model. Communities discovered by these methods therefore have interpretations in terms of statistical signficance. In this paper, we introduce a null for weighted networks called the continuous configuration model. We use the model both as a tool for community detection and for simulating weighted networks with null nodes. First, we propose a community extraction algorithm for weighted networks which incorporates iter…
Percolation based graph matching algorithms rely on the availability of seed vertex pairs as side information to efficiently match users across networks. Although such algorithms work well in practice, there are other types of side information available which are potentially useful to an attacker. In this paper, we consider the problem of matching two correlated graphs when an attacker has access to side information, either in the form of community labels or an imperfect initial matching. In the former case, we propose a naive graph matching algorithm by introducing the community degree vector…
We introduce a community detection method that finds clusters in network time-series by introducing an algorithm that finds significantly interconnected nodes across time. These connections are either increasing, decreasing, or constant over time. Significance of nodal connectivity within a set is judged using the Weighted Configuration Null Model at each time-point, then a novel significance-testing scheme is used to assess connectivity at all time points and the direction of its time-trend. We apply this method to bikeshare networks in New York City and Chicago and taxicab pickups and dropof…
Community detection is a key data analysis problem across different fields. During the past decades, numerous algorithms have been proposed to address this issue. However, most work on community detection does not address the issue of statistical significance. Although some research efforts have been made towards mining statistically significant communities, deriving an analytical solution of p-value for one community under the configuration model is still a challenging mission that remains unsolved. The configuration model is a widely used random graph model in community detection, in which t…
Community detection in subject-subject networks from psychometrics data
IMT School for Advanced Studies（IMT高等研究院）；Leiden University（莱顿大学）；INdAM-GNAMPA Istituto Nazionale di Alta Matematica ‘Francesco Severi’（意大利高等数学国家研究所（弗朗切斯科·塞韦里））；Lorentz Institute for Theoretical Physics, University of Leiden（莱顿大学洛伦兹理论物理研究所）
The analysis of stability of community structure is an important problem for scientists from many fields. Here, we propose a new framework to reveal hidden properties of community structure by quantitatively analyzing the dynamics of Potts model. Specifically we model the Potts procedure of community structure detection by a Markov process, which has a clear mathematical explanation. Critical topological information regarding to multivariate spin configuration could also be inferred from the spectral significance of the Markov process. We test our framework on some example networks and find it…
The problem of node-centric, or local, community detection in information networks refers to the identification of a community for a given input node, having limited information about the network topology. Existing methods for solving this problem, however, are not conceived to work on complex networks. In this paper, we propose a novel framework for local community detection based on the multilayer network model. Our approach relies on the maximization of the ratio between the community internal connection density and the external connection density, according to multilayer similarity-based c…
Nodes in real-world networks are usually organized in local modules. These groups, called communities, are intuitively defined as sub-graphs with a larger density of internal connections than of external links. In this work, we introduce a new measure aimed at quantifying the statistical significance of single communities. Extreme and Order Statistics are used to predict the statistics associated with individual clusters in random graphs. These distributions allows us to define one community significance as the probability that a generic clustering algorithm finds such a group in a random grap…
Community detection and link prediction are both of great significance in network analysis, which provide very valuable insights into topological structures of the network from different perspectives. In this paper, we propose a novel community detection algorithm with inclusion of link prediction, motivated by the question whether link prediction can be devoted to improving the accuracy of community partition. For link prediction, we propose two novel indices to compute the similarity between each pair of nodes, one of which aims to add missing links, and the other tries to remove spurious ed…
Current community detection algorithms operate by optimizing a statistic called modularity, which analyzes the distribution of positively weighted edges in a network. Modularity does not account for negatively weighted edges. This paper introduces a dual assortative modularity measure (DAMM) that incorporates both positively and negatively weighted edges. We describe the the DAMM statistic and illustrate its utility in a community detection algorithm. We evaluate the efficacy of the algorithm on both computer generated and real-world networks, showing that DAMM broadens the domain of networks …
The Normalized Mutual Information (NMI) has been widely used to evaluate the accuracy of community detection algorithms. However in this article we show that the NMI is seriously affected by systematic errors due to finite size of networks, and may give a wrong estimate of performance of algorithms in some cases. We give a simple theory to the finite-size effect of NMI and test our theory numerically. Then we propose a new metric for the accuracy of community detection, namely the relative Normalized Mutual Information (rNMI), which considers statistical significance of the NMI by comparing it…
Identifying communities (or clusters), namely groups of nodes with comparatively strong internal connectivity, is a fundamental task for deeply understanding the structure and function of a network. Yet, there is a lack of formal criteria for defining communities and for testing their significance. We propose a sharp definition which is based on a significance threshold. By means of a lumped Markov chain model of a random walker, a quality measure called "persistence probability" is associated to a cluster. Then the cluster is defined as an "$α$-community" if such a probability is not smaller …
Researchers use community-detection algorithms to reveal large-scale organization in biological and social networks, but community detection is useful only if the communities are significant and not a result of noisy data. To assess the statistical significance of the network communities, or the robustness of the detected structure, one approach is to perturb the network structure by removing links and measure how much the communities change. However, perturbing sparse networks is challenging because they are inherently sensitive; they shatter easily if links are removed. Here we propose a sim…
In many data sets, crucial information on the structure and temporality of a system coexists with noise and non-essential elements. In networked systems, for instance, some edges might be non-essential or exist only by chance. Filtering them out and extracting a set of relevant connections, the "network backbone", is a non-trivial task, and methods put forward until now do not address time-resolved networks, whose availability has strongly increased in recent years. We develop here such a method, by defining an adequate temporal network null model, which calculates the random chance of nodes t…
Identifying highly susceptible individuals in spreading processes is of great significance in controlling outbreaks. In this paper, we explore the susceptibility of people in susceptible-infectious-recovered (SIR) and rumor spreading dynamics. We first study the impact of community structure on people's susceptibility. Despite that the community structure can reduce the infected population given same infection rates, it will not deterministically affect nodes' susceptibility. We find the susceptibility of individuals is sensitive to the choice of spreading dynamics. For SIR spreading, since th…
Community detection in networks has drawn much attention in diverse fields, especially social sciences. Given its significance, there has been a large body of literature with approaches from many fields. Here we present a statistical framework that is representative, extensible, and that yields an estimator with good properties. Our proposed approach considers a stochastic blockmodel based on a logistic regression formulation with node correction terms. We follow a Bayesian approach that explicitly captures the community behavior via prior specification. We further adopt a data augmentation st…
To obtain the optimal number of communities is an important problem in detecting community structure. In this paper, we extend the measurement of community detecting algorithms to find the optimal community number. Based on the normalized mutual information index, which has been used as a measure for similarity of communities, a statistic $Ω(c)$ is proposed to detect the optimal number of communities. In general, when $Ω(c)$ reaches its local maximum, especially the first one, the corresponding number of communities \emph{c} is likely to be optimal in community detection. Moreover, the statist…
Starting from a general \textit{ansatz}, we show how community detection can be interpreted as finding the ground state of an infinite range spin glass. Our approach applies to weighted and directed networks alike. It contains the \textit{at hoc} introduced quality function from \cite{ReichardtPRL} and the modularity $Q$ as defined by Newman and Girvan \cite{Girvan03} as special cases. The community structure of the network is interpreted as the spin configuration that minimizes the energy of the spin glass with the spin states being the community indices. We elucidate the properties of the gr…
Multilayer networks are a useful way to capture and model multiple, binary or weighted relationships among a fixed group of objects. While community detection has proven to be a useful exploratory technique for the analysis of single-layer networks, the development of community detection methods for multilayer networks is still in its infancy. We propose and investigate a procedure, called Multilayer Extraction, that identifies densely connected vertex-layer sets in multilayer networks. Multilayer Extraction makes use of a significance based score that quantifies the connectivity of an observe…
Community detection is of fundamental significance for understanding the topology characters and the spreading dynamics on complex networks. While random walk is widely used and is proven effective in many community detection algorithms, there still exists two major defects: (i) the maximal length of random walk is too large to distinguish the clustering information if using the average step of all possible random walks; (ii) the useful community information at all other step lengths are missed if using a pre-assigned maximal length. In this paper, we propose a novel community detection method…
Many real-world networks such as the gene networks, protein-protein interaction networks and metabolic networks exhibit community structures, meaning the existence of groups of densely connected vertices in the networks. Many local similarity measures in the networks are closely related to the concept of the community structures, and may have positive effect on community detection in the networks. Here, various local similarity measures are used to extract the local structural information and then are applied to community detection in the networks by using the edge-reweighting strategy. The ef…
Community detection helps us simplify the complex configuration of networks, but communities are reliable only if they are statistically significant. To detect statistically significant communities, a common approach is to resample the original network and analyze the communities. But resampling assumes independence between samples, while the components of a network are inherently dependent. Therefore, we must understand how breaking dependencies between resampled components affects the results of the significance analysis. Here we use scientific communication as a model system to analyze this…
Given a graph of interactions, a module (also called a community or cluster) is a subset of nodes whose fitness is a function of the statistical significance of the pairwise interactions of nodes in the module. The topic of this paper is a model-based community finding approach, commonly referred to as modularity clustering, that was originally proposed by Newman and has subsequently been extremely popular in practice. Various heuristic methods are currently employed for finding the optimal solution. However, the exact computational complexity of this approach is still largely unknown. To this…
Complex networks analyses of many physical, biological and social phenomena show remarkable structural regularities, yet, their application in studying human past interaction remains underdeveloped. Here, we present an innovative method for identifying community structures in the archaeological record that allow for independent evaluation of the copper using societies in the Balkans, from c. 6200 to c. 3200 BC. We achieve this by exploring modularity of networked systems of these societies across an estimated 3000 years. We employ chemical data of copper-based objects from 79 archaeological si…
To understand how a complex system is organized and functions, researchers often identify communities in the system's network of interactions. Because it is practically impossible to explore all solutions to guarantee the best one, many community-detection algorithms rely on multiple stochastic searches. But for a given combination of network and stochastic algorithm, how many searches are sufficient to find a solution that is good enough? The standard approach is to pick a reasonably large number of searches and select the network partition with the highest quality or derive a consensus solut…
Computational models corresponding to supply security in natural gas networks aim to describe flows and consumption values in the case of component failures or unforseen pipeline shutdowns. The role of natural gas reservoires in this process has only been marginally analyzed in such models, and typically only on the level of countries. In this paper we define a computational framework, which is capable of interpreting real flow and reservoir data to assign a quantitative supply security related measure to reservoires, depending on how much the given reservoir is critical in the process of rest…
Parametric resampling schemes have been recently introduced in complex network analysis with the aim of assessing the statistical significance of graph clustering and the robustness of community partitions. We propose here a method to replicate structural features of complex networks based on the non-parametric resampling of the transition matrix associated with an unbiased random walk on the graph. We test this bootstrapping technique on synthetic and real-world modular networks and we show that the ensemble of replicates obtained through resampling can be used to improve the performance of s…
Given the increasing popularity of algorithms for overlapping clustering, in particular in social network analysis, quantitative measures are needed to measure the accuracy of a method. Given a set of true clusters, and the set of clusters found by an algorithm, these sets of clusters must be compared to see how similar or different the sets are. A normalized measure is desirable in many contexts, for example assigning a value of 0 where the two sets are totally dissimilar, and 1 where they are identical. A measure based on normalized mutual information, [1], has recently become popular. We de…
Algorithms for detecting clusters (including overlapping clusters) in graphs have received significant attention in the research community. A closely related important aspect of the problem -- quantification of statistical significance of overlap of clusters, remains relatively unexplored. This paper presents the first theoretical and practical results on quantifying statistically significant interactions between clusters in networks. Such problems commonly arise in diverse applications, ranging from social network analysis to systems biology. The paper addresses the problem of quantifying the…
Multilayer networks or MLNs (also called multiplexes or network of networks) are being used extensively for modeling and analysis of data sets with multiple entity and feature types as well as their relationships. As the concept of communities and hubs are used for these analysis, a structure-preserving definition for them on MLNs (that retains the original MLN structure and node/edge labels and types) and its efficient detection are critical. There is no structure-preserving definition of a community for a MLN as most of the current analyses aggregate a MLN to a single graph. Although there i…
Statistical significance of network clustering has been an unresolved problem since it was observed that community detection algorithms produce false positives even in random graphs. After a phase transition between undetectable and detectable cluster structures was discovered, the connection between spectra of adjacency matrices and detectability limits were shown, and both were calculated for a wide range of networks with arbitrary degree distributions and community structure. In practice the full eigenspectrum is not known, and whether a given network has any communities within detectabilit…
Background: Network communities help the functional organization and evolution of complex networks. However, the development of a method, which is both fast and accurate, provides modular overlaps and partitions of a heterogeneous network, has proven to be rather difficult. Methodology/Principal Findings: Here we introduce the novel concept of ModuLand, an integrative method family determining overlapping network modules as hills of an influence function-based, centrality-type community landscape, and including several widely used modularization methods as special cases. As various adaptations…
The modern science of networks has brought significant advances to our understanding of complex systems. One of the most relevant features of graphs representing real systems is community structure, or clustering, i. e. the organization of vertices in clusters, with many edges joining vertices of the same cluster and comparatively few edges joining vertices of different clusters. Such clusters, or communities, can be considered as fairly independent compartments of a graph, playing a similar role like, e. g., the tissues or the organs in the human body. Detecting communities is of great import…
Smart resilience is the beneficial result of the collision course of the fields of data science and urban resilience to flooding. The objective of this study is to propose and demonstrate a smart flood resilience framework that leverages heterogeneous community-scale big data and infrastructure sensor data to enhance predictive risk monitoring and situational awareness. The smart flood resilience framework focuses on four core capabilities that could be augmented by the use of heterogeneous community-scale big data and analytics techniques: (1) predictive flood risk mapping; (2) automated rapi…
Community detection and edge prediction are both forms of link mining: they are concerned with discovering the relations between vertices in networks. Some of the vertex similarity measures used in edge prediction are closely related to the concept of community structure. We use this insight to propose a novel method for improving existing community detection algorithms by using a simple vertex similarity measure. We show that this new strategy can be more effective in detecting communities than the basic community detection algorithms.
Use of the Hirsch-index ($h$) as measure of an author's visibility in the scientific literature has become popular as an alternative to a gross measure like total citations (c). I show that, at least in astrophysics, $h$ correlates tightly with overall citations. The mean relation is $h=0.5(\sqrt c+1)$. Outliers are few and not too far from the mean, especially if `normalized' ADS citations are used for $c$ and $h$. Whatever the theoretical reasoning behind it, the Hirsch index in practice does not appear to measure something significantly new.
Email service providers have employed many email classification and prioritization systems over the last decade to improve their services. In order to assist email services, we propose a personalized email community detection method to discover the groupings of email users based on their structural and semantic intimacy. We extract the personalized social graph from a set of emails by uniquely leveraging each node with communication behavior. Subsequently, collaborative similarity measure (CSM) based intra-graph clustering approach detects personalized communities. The empirical analysis shows…
Many complex networks exhibit a modular structure of densely connected groups of nodes. Usually, such a modular structure is uncovered by the optimization of some quality function. Although flawed, modularity remains one of the most popular quality functions. The Louvain algorithm was originally developed for optimizing modularity, but has been applied to a variety of methods. As such, speeding up the Louvain algorithm, enables the analysis of larger graphs in a shorter time for various methods. We here suggest to consider moving nodes to a random neighbor community, instead of the best neighb…
Modularity is widely used to effectively measure the strength of the disjoint community structure found by community detection algorithms. Although several overlapping extensions of modularity were proposed to measure the quality of overlapping community structure, there is lack of systematic comparison of different extensions. To fill this gap, we overview overlapping extensions of modularity to select the best. In addition, we extend the Modularity Density metric to enable its usage for overlapping communities. The experimental results on four real networks using overlapping extensions of mo…
Community detection can be considered as a variant of cluster analysis applied to complex networks. For this reason, all existing studies have been using tools derived from this field when evaluating community detection algorithms. However, those are not completely relevant in the context of network analysis, because they ignore an essential part of the available information: the network structure. Therefore, they can lead to incorrect interpretations. In this article, we review these measures, and illustrate this limitation. We propose a modification to solve this problem, and apply it to the…
Modularity is widely used to effectively measure the strength of the community structure found by community detection algorithms. However, modularity maximization suffers from two opposite yet coexisting problems: in some cases, it tends to favor small communities over large ones while in others, large communities over small ones. The latter tendency is known in the literature as the resolution limit problem. To address them, we propose to modify modularity by subtracting from it the fraction of edges connecting nodes of different communities and by including community density into modularity.…
Modularity, since its introduction, has remained one of the most widely used metrics to assess the quality of community structure in a complex network. However the resolution limit problem associated with modularity limits its applicability to networks with community sizes smaller than a certain scale. In the past various attempts have been made to solve this problem. More recently a new metric, modularity density, was introduced for the quality of community structure in networks in order to solve some of the known problems with modularity, particularly the resolution limit problem. Modularity…
Community structure is one of the most important properties of networks. Most community algorithms are not suitable for large networks because of their time consuming. In fact there are lots of networks with millons even billons of nodes. In such case, most algorithms running in time O(n2logn) or even larger are not practical. What we need are linear or approximately linear time algorithm. Rising in response to such needs, we propose a quick methods to evaluate community structure in networks and then put forward a local community algorithm with nearly linear time based on random walks. Using …
Understanding the network structure, and finding out the influential nodes is a challenging issue in the large networks. Identifying the most influential nodes in the network can be useful in many applications like immunization of nodes in case of epidemic spreading, during intentional attacks on complex networks. A lot of research is done to devise centrality measures which could efficiently identify the most influential nodes in the network. There are two major approaches to the problem: On one hand, deterministic strategies that exploit knowledge about the overall network topology in order …
Certain crimes are hardly committed by individuals but carefully organised by group of associates and affiliates loosely connected to each other with a single or small group of individuals coordinating the overall actions. A common starting point in understanding the structural organisation of criminal groups is to identify the criminals and their associates. Situations arise in many criminal datasets where there is no direct connection among the criminals. In this paper, we investigate ties and community structure in crime data in order to understand the operations of both traditional and cyb…
The information theoretic quantity known as mutual information finds wide use in classification and community detection analyses to compare two classifications of the same set of objects into groups. In the context of classification algorithms, for instance, it is often used to compare discovered classes to known ground truth and hence to quantify algorithm performance. Here we argue that the standard mutual information, as commonly defined, omits a crucial term which can become large under real-world conditions, producing results that can be substantially in error. We demonstrate how to corre…
Based on an expert systems approach, the issue of community detection can be conceptualized as a clustering model for networks. Building upon this further, community structure can be measured through a clustering coefficient, which is generated from the number of existing triangles around the nodes over the number of triangles that can be hypothetically constructed. This paper provides a new definition of the clustering coefficient for weighted networks under a generalized definition of triangles. Specifically, a novel concept of triangles is introduced, based on the assumption that, should th…
This report presents a very simple algorithm for overlaping community-detection in large graphs under constraints such as the minimum and maximum number of members allowed. The algorithm is based on the simulation of random walks and measures the entropy of each random walk to detect the discovery of a community.
