arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2405.00764cs.DBcs.DM
收藏

利用离散优化改进数据清洗

Improving Data Cleaning Using Discrete Optimization

Kenneth Smith, Sharlee Climer

AI总结:

本文通过将整数规划重构为线性规划并优化变量与并行化,扩展 Mr. Clean 算法以改进缺失数据删除,在真实数据集上实现更高的数据保留率和更低运行时间。

摘要:

在任何分析流程中,最重要的处理步骤之一是处理缺失数据。传统方法直接删除任何含有缺失元素的样本或特征。近期的填补方法则基于观测数据与缺失元素之间的假设关系来替换缺失数据。然而,在这两种极端方法之间,存在一种在很大程度上未被充分探索的替代方案。部分删除方法根据用户定义的标准,移除过量的缺失数据。它们可以替代传统删除方法,也可以作为填补之前的预处理步骤。在本文中,我们扩展了 Mr. Clean 算法套件,重点关注所有缺失数据均被移除的场景。我们证明 RowCol 整数规划可以被重构为线性规划,从而降低运行时间。此外,Element 整数规划可以被重新表述,以减少变量数量并支持高度并行化。利用来自遗传学、基因表达和单细胞 RNA 测序实验的真实数据集,我们证明我们的算法在多个缺失率条件下优于现有的删除技术,在运行时间和数据保留之间取得平衡。我们的组合贪心算法在 150 个场景中的 126 个中保留了最大数量的有效元素,并在其余实验中的 23 个中保持在最大值的 1% 以内。重新表述的 Element IP 在移除所有缺失数据时与贪心算法互为补充,与其通用对应方法相比,在较大数据集上具有更短的运行时间和更多的有效元素。这两个程序相比传统删除技术大幅增加了保留的有效数据量,并进一步改进了现有的部分删除算法。

英文摘要:

One of the most important processing steps in any analysis pipeline is handling missing data. Traditional approaches simply delete any sample or feature with missing elements. Recent imputation methods replace missing data based on assumed relationships between observed data and the missing elements. However, there is a largely under-explored alternative amid these extremes. Partial deletion approaches remove excessive amounts of missing data, as defined by the user. They can be used in place of traditional deletion or as a precursor to imputation. In this manuscript, we expand upon the Mr. Clean suite of algorithms, focusing on the scenario where all missing data is removed. We show that the RowCol Integer Program can be recast as a Linear Program, thereby reducing runtime. Additionally, the Element Integer Program can be reformulated to reduce the number of variables and allow for high levels of parallelization. Using real-world data sets from genetic, gene expression, and single cell RNA-seq experiments we demonstrate that our algorithms outperform existing deletion techniques over several missingness values, balancing runtime and data retention. Our combined greedy algorithm retains the maximum number of valid elements in 126 of 150 scenarios and stays within 1\% of maximum in 23 of the remaining experiments. The reformulated Element IP complements the greedy algorithm when removing all missing data, boasting a reduced runtime and increase in valid elements in larger data sets, over its generic counterpart. These two programs greatly increase the amount of valid data retained over traditional deletion techniques and further improve on existing partial deletion algorithms.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑