分类程序足够好吗？分类学习的拟合优度评估工具
Is a Classification Procedure Good Enough? A Goodness-of-Fit Assessment Tool for Classification Learning
摘要：
近年来，随机森林、Boosting、神经网络等非传统的分类方法得到了广泛的应用。它们的性能通常以分类准确性来衡量。虽然分类错误率等很重要，但它们并没有解决一个基本问题：分类方法是否不适合？据我们所知，目前还没有一种方法可以评估一般分类程序的拟合优度。事实上，缺乏参数假设使得构建适当的测试具有挑战性。为了克服这个困难，我们提出了一种称为BAGofT的方法，将数据分为训练集和验证集。首先，将评估的分类过程应用于训练集，该训练集也用于自适应地找到揭示最严重的欠拟合区域的数据分组。然后，基于此分组，我们通过比较估计的成功概率和来自验证集的实际观察响应来计算检验统计量。数据分割保证了检验的大小在原假设下得到控制，并且在备择假设下，随着样本量的增加，检验的功效变为1。对于测试参数分类模型，BAGofT具有比现有方法更宽的范围，因为它不限于特定的参数模型（例如，逻辑斯谛回归）。广泛的模拟研究表明，效用的BAGofT时，评估一般的分类程序和它的优势，一些现有的方法时，测试参数分类模型。
英文摘要：
In recent years, many non-traditional classification methods, such as Random Forest, Boosting, and neural network, have been widely used in applications. Their performance is typically measured in terms of classification accuracy. While the classification error rate and the like are important, they do not address a fundamental question: Is the classification method underfitted? To our best knowledge, there is no existing method that can assess the goodness-of-fit of a general classification procedure. Indeed, the lack of a parametric assumption makes it challenging to construct proper tests. To overcome this difficulty, we propose a methodology called BAGofT that splits the data into a training set and a validation set. First, the classification procedure to assess is applied to the training set, which is also used to adaptively find a data grouping that reveals the most severe regions of underfitting. Then, based on this grouping, we calculate a test statistic by comparing the estimated success probabilities and the actual observed responses from the validation set. The data splitting guarantees that the size of the test is controlled under the null hypothesis, and the power of the test goes to one as the sample size increases under the alternative hypothesis. For testing parametric classification models, the BAGofT has a broader scope than the existing methods since it is not restricted to specific parametric models (e.g., logistic regression). Extensive simulation studies show the utility of the BAGofT when assessing general classification procedures and its strengths over some existing methods when testing parametric classification models.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
