Many existing two-phase kernel-based hypothesis transfer learning algorithms employ the same kernel regularization across phases and rely on the known smoothness of functions to obtain optimality. Therefore, they fail to adapt to the varying and unknown smoothness between the target/source and their offset in practice. In this paper, we address these problems by proposing Smoothness Adaptive Transfer Learning (SATL), a two-phase kernel ridge regression(KRR)-based algorithm. We first prove that employing the misspecified fixed bandwidth Gaussian kernel in target-only KRR learning can achieve minimax optimality and derive an adaptive procedure to the unknown Sobolev smoothness. Leveraging these results, SATL employs Gaussian kernels in both phases so that the estimators can adapt to the unknown smoothness of the target/source and their offset function. We derive the minimax lower bound of the learning problem in excess risk and show that SATL enjoys a matching upper bound up to a logarithmic factor. The minimax convergence rate sheds light on the factors influencing transfer dynamics and demonstrates the superiority of SATL compared to non-transfer learning settings. While our main objective is a theoretical analysis, we also conduct several experiments to confirm our results.
A simple parameter-free and adaptive approach to optimization under a minimal local smoothness assumption
University of California, Berkeley(加州大学伯克利分校);Noah’s Ark Lab, Huawei Technologies(华为诺亚方舟实验室);INRIA Lille - Nord Europe(法国国家信息与自动化研究所里尔-北欧分部)
研究了在\emph{budgeted number of evaluations}下函数的优化问题。我们只假设函数在其全局最优点附近是\emph{locally}光滑的。优化的难度根据1)函数评估的\emph{noise} $b$的量和2)函数的局部平滑度$d$来测量。较小的$d$导致较小的优化误差。我们采用了一种新的、简单的、无参数的方法。首先,对于$b$和$d$的所有值,这种方法至少恢复了最先进的遗憾保证。其次,我们的方法还获得了这些结果,同时\textit{agnostic}的值都$b$和$d$。这导致第一算法自然地适应于噪声$b$的\textit{unknown}范围,并且导致在中等和低噪声范围中的显著改进。第三,我们的方法也获得了显着的改善,在国家的最先进的SOO算法时,噪声是非常低的,其中包括确定性反馈下的优化($b=0$)的情况下。在那里,在我们的最小局部平滑假设下,这种改进是指数量级的,并且适用于一类函数,该函数涵盖了从业者优化的绝大多数函数($d=0$)。我们表明,我们的算法改进证明了在实验中,我们凭经验表明,在共同的基准更快的收敛。