arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2110.00568cs.LGstat.ML
收藏

条件深度高斯过程:经验贝叶斯超数据学习

Conditional Deep Gaussian Processes: empirical Bayes hyperdata learning

Rutgers University(罗格斯大学);Institute for Advanced Studies(高等研究院)

Chi-Ken Lu, Patrick Shafto

摘要:

希望将深度学习的表达能力与高斯过程(GP)结合在一个表达性贝叶斯学习模型中。深度内核学习成功地采用了深度网络进行特征提取,然后将GP用作函数模型。最近,有人建议,虽然训练与边际似然,特征提取器的确定性性质可能会导致过度拟合,而用贝叶斯网络的替代似乎治愈it.Here,我们提出了条件深高斯过程(DGP),其中层次组成的中间GP由超数据支持,暴露的GP保持零均值。受稀疏GP中诱导点的启发,超数据也扮演函数支持的角色,但是超参数而不是随机变量。我们按照我们以前的时刻匹配的方法来近似的边缘先验条件DGP的GP携带一个有效的内核。因此,与经验贝叶斯一样,超数据是通过优化近似边际似然来学习的,近似边际似然经由内核隐式地依赖于超数据。我们将在潜在空间中的稠密超数据的限制下证明与深度核学习的等价性。然而,条件DGP和相应的近似推理比深度内核学习更具有贝叶斯性。初步的外推结果表明,通过利用精确的协方差和超数据学习,与GP内核组成,DGP变分推理和深度内核学习相比,从层次结构的深度表现出强大的表达能力。我们还解决了我们的模型的非高斯方面,以及升级到一个完整的贝叶斯推理的方式。

英文摘要:

It is desirable to combine the expressive power of deep learning with Gaussian Process (GP) in one expressive Bayesian learning model. Deep kernel learning showed success in adopting a deep network for feature extraction followed by a GP used as function model. Recently,it was suggested that, albeit training with marginal likelihood, the deterministic nature of feature extractor might lead to overfitting while the replacement with a Bayesian network seemed to cure it. Here, we propose the conditional Deep Gaussian Process (DGP) in which the intermediate GPs in hierarchical composition are supported by the hyperdata and the exposed GP remains zero mean. Motivated by the inducing points in sparse GP, the hyperdata also play the role of function supports, but are hyperparameters rather than random variables. We follow our previous moment matching approach to approximate the marginal prior for conditional DGP with a GP carrying an effective kernel. Thus, as in empirical Bayes, the hyperdata are learned by optimizing the approximate marginal likelihood which implicitly depends on the hyperdata via the kernel. We shall show the equivalence with the deep kernel learning in the limit of dense hyperdata in latent space. However, the conditional DGP and the corresponding approximate inference enjoy the benefit of being more Bayesian than deep kernel learning. Preliminary extrapolation results demonstrate expressive power from the depth of hierarchy by exploiting the exact covariance and hyperdata learning, in comparison with GP kernel composition, DGP variational inference and deep kernel learning. We also address the non-Gaussian aspect of our model as well as way of upgrading to a full Bayes inference.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑