通用坚持：防止自适应数据科学中的错误发现
The Generic Holdout: Preventing False-Discoveries in Adaptive Data Science
摘要：
自适应数据分析对科学提出了挑战，因为它能够在中等规模的数据集上产生错误的假设。通常，对于非自适应数据分析（其中生成对数据的查询而不受对先前查询的回答的影响），包含$n$样本的数据集可以支持$n$中的指数级的许多查询。这个数字在天真的自适应数据分析下线性减少到许多，甚至复杂的补救措施，如可重用的保持（Dwork et. al 2015）仅允许在$n$中进行二次多个查询。在这项工作中，我们提出了一个适应性科学的新框架，它在有限但科学相关的设置下以指数方式提高了查询的数量，其中科学家的目标是根据样本找到关于宇宙的单个（或几个）真实假设。这样的设置可以描述一些疾病的预测因素的基础上的医疗数据，分析师可能希望尝试一些预测模型，直到一个令人满意的是found. We的解决方案，通用的保持，包括两个简单的成分：（1）的数据划分成一个探索集和一个保持集和（2）一个有限的曝光策略的保持集。分析师可以随意使用探索集，但是当针对坚持集检验假设时，分析师只知道这个问题的答案：“给定的假设在坚持集上是真的吗？--而且没有更多的信息，比如假设与坚持者集合的拟合程度。由此产生的方案是立即分析，但尽管它的简单性，我们不相信我们的方法是显而易见的，在实践中的许多违规行为证明。我们的建议可以被视为预注册的替代方案，并允许研究人员获得自适应数据分析的好处，而没有自适应性的问题。
英文摘要：
Adaptive data analysis has posed a challenge to science due to its ability to generate false hypotheses on moderately large data sets. In general, with non-adaptive data analyses (where queries to the data are generated without being influenced by answers to previous queries) a data set containing $n$ samples may support exponentially many queries in $n$. This number reduces to linearly many under naive adaptive data analysis, and even sophisticated remedies such as the Reusable Holdout (Dwork et. al 2015) only allow quadratically many queries in $n$. In this work, we propose a new framework for adaptive science which exponentially improves on this number of queries under a restricted yet scientifically relevant setting, where the goal of the scientist is to find a single (or a few) true hypotheses about the universe based on the samples. Such a setting may describe the search for predictive factors of some disease based on medical data, where the analyst may wish to try a number of predictive models until a satisfactory one is found. Our solution, the Generic Holdout, involves two simple ingredients: (1) a partitioning of the data into a exploration set and a holdout set and (2) a limited exposure strategy for the holdout set. An analyst is free to use the exploration set arbitrarily, but when testing hypotheses against the holdout set, the analyst only learns the answer to the question: "Is the given hypothesis true (empirically) on the holdout set?" -- and no more information, such as "how well" the hypothesis fits the holdout set. The resulting scheme is immediate to analyze, but despite its simplicity we do not believe our method is obvious, as evidenced by the many violations in practice. Our proposal can be seen as an alternative to pre-registration, and allows researchers to get the benefits of adaptive data analysis without the problems of adaptivity.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
