arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2210.16405cs.LG
收藏

分类生成模型的评价--弥合真实数据与合成数据之间的差距

Evaluation of Categorical Generative Models -- Bridging the Gap Between Real and Synthetic Data

McGill University(麦吉尔大学)

Florence Regol, Anja Kroon, Mark Coates

摘要:

机器学习社区主要依靠真实数据来对算法进行基准测试,因为它提供了模型适用性的令人信服的证据。对合成数据集的评估是一个强大的工具,可以更好地了解模型的优势,劣势和整体能力。获得这些见解对于生成式建模尤其重要,因为目标量完全未知。在文献中已经报道了与生成模型的评估相关的多个问题。我们认为,这些问题可以通过基于地面真相的评估来避免。对合成实验的普遍批评是,它们过于简化,不能代表实际情况。因此,我们的实验设置是针对一个现实的生成任务。我们专注于分类数据,并介绍了一个适当的可扩展的评估方法。我们的方法涉及到任务生成模型学习在高维环境中的分布。然后,我们依次对大空间进行分组,以获得更小的概率空间,其中可以应用有意义的统计测试。我们考虑越来越大的概率空间,这对应于越来越困难的建模任务,并根据它们在被检测到与地面事实相差太远之前可以达到的最高任务难度来比较生成模型。我们验证了我们的评估过程与合成生成模型和当前国家的最先进的分类生成模型的合成实验。

英文摘要:

The machine learning community has mainly relied on real data to benchmark algorithms as it provides compelling evidence of model applicability. Evaluation on synthetic datasets can be a powerful tool to provide a better understanding of a model's strengths, weaknesses, and overall capabilities. Gaining these insights can be particularly important for generative modeling as the target quantity is completely unknown. Multiple issues related to the evaluation of generative models have been reported in the literature. We argue those problems can be avoided by an evaluation based on ground truth. General criticisms of synthetic experiments are that they are too simplified and not representative of practical scenarios. As such, our experimental setting is tailored to a realistic generative task. We focus on categorical data and introduce an appropriately scalable evaluation method. Our method involves tasking a generative model to learn a distribution in a high-dimensional setting. We then successively bin the large space to obtain smaller probability spaces where meaningful statistical tests can be applied. We consider increasingly large probability spaces, which correspond to increasingly difficult modeling tasks and compare the generative models based on the highest task difficulty they can reach before being detected as being too far from the ground truth. We validate our evaluation procedure with synthetic experiments on both synthetic generative models and current state-of-the-art categorical generative models.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑