发表机构
University of California, Irvine（加州大学欧文分校）
机构由 AI 辅助整理，请以论文原文为准。AI 中文总结
本文针对高斯过程超参数优化场景，评估最大似然估计的脆弱性，提出实用解决方案，其在贝叶斯优化等任务中有效，构建的高斯过程可在多方面超越表格基础模型。
AI 中文摘要
机器学习（ML）已成为现代工程设计工作流程中不可或缺的一部分。训练ML模型的关键步骤是选择损失函数，这可通过最大似然估计（MLE）、交叉验证等多种技术系统地构建。尽管MLE是训练ML模型最流行、有效且直观的机制之一，但它具有脆弱性：若其支撑假设未被满足，训练出的ML模型可能泛化能力差。这种脆弱性甚至影响到广泛应用于工程设计、常被（错误地认为）对过拟合非常鲁棒的高斯过程（GPs）。本文中，我们针对用于概率回归或分类任务的GPs训练场景，从根本上评估MLE的脆弱性，将基于理论的指标与MLE进行比较，并提出实用解决方案。我们的大量研究表明，所提解决方案在贝叶斯优化等下游设计任务中有效，为从业者构建准确且鲁棒的GPs提供了蓝图，这类GPs在预测精度、不确定性量化和推理成本方面甚至可超越表格基础模型。我们的贡献已通过GitHub公开提供，链接为https://this.URL。
英文摘要
Machine learning (ML) has become an indispensable part of modern engineering design workflows. A crucial step in training an ML model is the selection of the loss function which can be systematically formulated via various techniques such as maximum likelihood estimation (MLE) and cross-validation . While MLE is one of the most popular, effective, and intuitive mechanisms for training ML models, it is brittle: if the assumptions underpinning it are not met, the trained ML model may generalize poorly. This brittleness affects even Gaussian processes (GPs) which are widely used in engineering design and are often (incorrectly) presumed to be very robust to overfitting. In this paper, we fundamentally evaluate the brittleness of MLE in the context of training GPs for probabilistic regression or classification tasks. We compare theoretically grounded metrics against MLE and propose practical solutions. Our extensive studies demonstrate the effectiveness of our solutions in downstream design tasks such as Bayesian optimization and provide a blueprint for practitioners to build accurate and robust GPs that can even outperform tabular foundation models in terms of prediction accuracy, uncertainty quantification, and inference cost. Our contributions are publicly available via GitHub at https://github.com/Bostanabad-Research-Group/GP-vs-TabPFN-vs-GPyTorch.
Comments26 pages, 10 figures, 2 tables
