arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2410.09239cs.LGstat.ML
收藏

通过潜在Kronecker结构扩展高斯过程以用于学习曲线预测

Scaling Gaussian Processes for Learning Curve Prediction via Latent Kronecker Structure

Meta;University of Cambridge(剑桥大学);Max Planck Institute for Intelligent Systems(马克斯·普朗克智能系统研究所)

Jihao Andreas Lin, Sebastian Ament, Maximilian Balandat, Eytan Bakshy

AI总结:

提出潜在Kronecker结构的高斯过程方法,在处理缺失值的同时降低计算复杂度,并在学习曲线预测任务上达到Transformer的性能。

摘要:

AutoML中的一项关键任务是将机器学习模型的学习曲线作为模型超参数和训练进程的函数进行联合建模。虽然高斯过程(GPs)适合此任务,但朴素GPs在处理n个超参数配置和每个超参数O(m)个学习曲线观测值时,需要O(n^3 m^3)的时间和O(n^2 m^2)的空间。通过Kronecker结构进行的高效推断通常由于缺失学习曲线值而与早停机制不兼容。我们引入潜在Kronecker结构,以利用高效的乘积核,同时处理缺失值。具体而言,我们将观测值的联合协方差矩阵解释为潜在Kronecker乘积的投影。结合迭代线性求解器和结构化矩阵向量乘法,我们的方法仅需O(n^3 + m^3)的时间和O(n^2 + m^2)的空间。我们表明,我们的GP模型在学习曲线预测任务上能够匹配Transformer的性能。

英文摘要:

A key task in AutoML is to model learning curves of machine learning models jointly as a function of model hyper-parameters and training progression. While Gaussian processes (GPs) are suitable for this task, naïve GPs require $\mathcal{O}(n^3m^3)$ time and $\mathcal{O}(n^2 m^2)$ space for $n$ hyper-parameter configurations and $\mathcal{O}(m)$ learning curve observations per hyper-parameter. Efficient inference via Kronecker structure is typically incompatible with early-stopping due to missing learning curve values. We impose $\textit{latent Kronecker structure}$ to leverage efficient product kernels while handling missing values. In particular, we interpret the joint covariance matrix of observed values as the projection of a latent Kronecker product. Combined with iterative linear solvers and structured matrix-vector multiplication, our method only requires $\mathcal{O}(n^3 + m^3)$ time and $\mathcal{O}(n^2 + m^2)$ space. We show that our GP model can match the performance of a Transformer on a learning curve prediction task.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑