arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2009.09471stat.APcs.DBcs.LGstat.ML
收藏

SYNC:一个基于Copula的聚合数据生成框架

SYNC: A Copula based Framework for Generating Synthetic Data from Aggregated Sources

Northeastern University Toronto(多伦多东北大学);Arima Inc.(阿里马公司);H. John Heinz III College Carnegie Mellon University(卡内基梅隆大学约翰·海因茨三世学院);University of Toronto(多伦多大学)

Zheng Li, Yue Zhao, Jialin Fu

摘要:

合成数据集是以编程方式生成的数据对象,当直接收集困难或成本高昂时,从多个源创建单个数据集可能很有价值。虽然它是许多数据科学任务的基本步骤,但缺乏有效和标准的框架。在本文中,我们研究了一种称为降尺度的特定合成数据生成任务,这是一种推断高分辨率,难以收集的信息(例如,个人水平的记录)从许多低分辨率,易于收集的来源,并提出了一个多阶段的框架,称为同步(通过高斯Copula合成数据生成)。对于给定的低分辨率数据集,SYNC的中心思想是将高斯copula模型拟合到每个低分辨率数据集,以便正确地捕获依赖性和边缘分布,然后从拟合的模型中采样以获得所需的高分辨率子集。然后使用预测模型将采样子集合并为一个,最后根据低分辨率边缘约束对采样数据集进行缩放。我们在这项工作中做出了四项重要贡献:1)提出一种新的框架,用于通过结合最先进的机器学习和统计技术从聚合数据源生成个体级数据,2)执行模拟研究以验证SYNC作为合成数据生成算法的性能,3)证明其作为特征工程工具的价值,以及在难以通过两个真实世界数据集收集的情况下的数据收集的替代方案,4)发布易于使用的框架实现,以在生产级别上实现可再现性和可扩展性,从而容易地合并新数据。

英文摘要:

A synthetic dataset is a data object that is generated programmatically, and it may be valuable to creating a single dataset from multiple sources when direct collection is difficult or costly. Although it is a fundamental step for many data science tasks, an efficient and standard framework is absent. In this paper, we study a specific synthetic data generation task called downscaling, a procedure to infer high-resolution, harder-to-collect information (e.g., individual level records) from many low-resolution, easy-to-collect sources, and propose a multi-stage framework called SYNC (Synthetic Data Generation via Gaussian Copula). For given low-resolution datasets, the central idea of SYNC is to fit Gaussian copula models to each of the low-resolution datasets in order to correctly capture dependencies and marginal distributions, and then sample from the fitted models to obtain the desired high-resolution subsets. Predictive models are then used to merge sampled subsets into one, and finally, sampled datasets are scaled according to low-resolution marginal constraints. We make four key contributions in this work: 1) propose a novel framework for generating individual level data from aggregated data sources by combining state-of-the-art machine learning and statistical techniques, 2) perform simulation studies to validate SYNC's performance as a synthetic data generation algorithm, 3) demonstrate its value as a feature engineering tool, as well as an alternative to data collection in situations where gathering is difficult through two real-world datasets, 4) release an easy-to-use framework implementation for reproducibility and scalability at the production level that easily incorporates new data.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑