哪些泄漏类型重要?2047个基准数据集的定量景观
Which Leakage Types Matter? A Quantitative Landscape Across 2,047 Benchmark Datasets
AI总结:
通过在2047个独立同分布表格数据集上进行28项受试者内反事实实验,以及129个时间序列数据集的边界实验,定量评估了机器学习中四类数据泄漏的严重性。
摘要:
通过在2047个独立同分布表格数据集上进行28项受试者内反事实实验,以及129个时间序列数据集的边界实验,测量了机器学习中四类数据泄漏的严重程度。第一类(估计:在全数据上拟合缩放器)可忽略:所有九种条件产生的$|ΔAUC| \leq 0.005$。第二类(选择:偷窥、种子挑选)影响显著:测量效果与约90%的噪声利用导致报告分数膨胀一致。第三类(记忆)随模型容量增加:在10%重复时,$d_z$从0.37(朴素贝叶斯)到1.11(决策树)。第四类(边界)在随机交叉验证下不可见。在这个独立同分布表格数据体制中,教科书的重点被颠覆:归一化泄漏最不重要;而实际数据集规模下的选择泄漏最为重要。
英文摘要:
Twenty-eight within-subject counterfactual experiments across 2,047 iid tabular datasets, plus a boundary experiment on 129 temporal datasets, measure the severity of four data leakage classes in machine learning. Class I (estimation: fitting scalers on full data) is negligible: all nine conditions produce $|ΔAUC| \leq 0.005$. Class II (selection: peeking, seed cherry-picking) is substantial: the measured effect is consistent with about 90% noise exploitation inflating reported scores. Class III (memorization) scales with model capacity: $d_z$ = 0.37 (Naive Bayes) to 1.11 (Decision Tree) at 10% duplication. Class IV (boundary) is invisible under random cross-validation. Within this iid tabular regime, the textbook emphasis is inverted: normalization leakage matters least; selection leakage at practical dataset sizes matters most.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。