在大数据时代,信息集成往往需要从海量数据源中提取丰富的数据。由于数据源的数量众多,数据源的选择在信息集成中起着至关重要的作用,因为访问所有的数据源是昂贵的,甚至是不可能的。数据源的选择应考虑效率和有效性问题。为了提高效率,该方法应实现高性能和可扩展性,以适应大数据源量。从有效性方面来看,应考虑数据质量和来源重叠问题,因为数据质量因数据来源而异,所提供数据的准确性和覆盖面有很大差异,来源重叠甚至会降低从选定数据来源整合的数据质量。本文研究了\textit{Big Data Era}中的数据源选择问题,提出了一种可以扩展到数百万数据源的数据集并保证结果质量的方法。出于这一动机,我们提出了一个新的目标函数,以期望的真值数的源可以提供作为一个标准,以评估数据源的贡献。基于我们提出的索引,我们提出了一个可扩展的算法和两个剪枝策略,以提高效率,而不牺牲精度。在真实数据集和合成数据集上的实验结果表明,该方法能够有效地选择出真实值占较大比例的数据源,并且能够扩展到海量数据源。
英文摘要:
In Big data era, information integration often requires abundant data extracted from massive data sources. Due to a large number of data sources, data source selection plays a crucial role in information integration, since it is costly and even impossible to access all data sources. Data Source selection should consider both efficiency and effectiveness issues. For efficiency, the approach should achieve high performance and be scalability to fit large data source amount. From effectiveness aspect, data quality and overlapping of sources are to be considered, since data quality varies much from data sources, with significant differences in the accuracy and coverage of the data provided, and the overlapping of sources can even lower the quality of data integrated from selected data sources.
In this paper, we study source selection problem in \textit{Big Data Era} and propose methods which can scale to datasets with up to millions of data sources and guarantee the quality of results. Motivated by this, we propose a new object function taking the expected number of true values a source can provide as a criteria to evaluate the contribution of a data source. Based on our proposed index we present a scalable algorithm and two pruning strategies to improve the efficiency without sacrificing precision. Experimental results on both real world and synthetic data sets show that our methods can select sources providing a large proportion of true values efficiently and can scale to massive data sources.
Spatial Heterogeneity, Scale, Data Character, and Sustainable Transport in the Big Data Era
鉴于大数据的出现,我倡导并主张从托布勒定律到缩放定律,从欧几里德几何到分形几何,从高斯统计到帕累托统计,更重要的是从笛卡尔的机械思维到亚历山大的有机思维的范式转变。分形几何属于分形的第三种定义-也就是说,如果比大事物更多的小事物的缩放多次重复出现,则集合或图案是分形的(Jiang and Yin 2014)-而不是分形的第二种定义,它需要尺度和细节之间的幂律(Mandelbrot 1982)。新的分形几何更倾向于“遵循规则,约束和偶然条件,这些都是不可避免地在现实世界中遇到的”(Alexander et al. 2012,p. 395),不仅用于理解复杂性,而且用于创建复杂或有生命的结构(Alexander 2002-2005)。这篇社论试图澄清为什么范式转变是必不可少的,并详细阐述了几个概念,包括空间异质性(缩放定律),规模(或规模的第四种含义),数据特征(相对于数据质量),以及大数据时代的可持续交通。