arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2402.09327cs.LG
收藏

随机凸优化的信息复杂度:对泛化与记忆的应用

Information Complexity of Stochastic Convex Optimization: Applications to Generalization and Memorization

Ben-Gurion University(本-古里安大学);Vector Institute(矢量研究所);Google DeepMind(谷歌DeepMind);Northeastern University(东北大学);Tel Aviv University(特拉维夫大学);University of Toronto(多伦多大学)

Idan Attias, Gintare Karolina Dziugaite, Mahdi Haghifam, Roi Livni, Daniel M. Roy

AI总结:

本研究刻画了随机凸优化中学习算法准确度与条件互信息(记忆度量)之间的权衡,证明了误差下界的依赖关系,并展示了记忆的关键作用及对泛化界限的影响。

摘要:

在本工作中,我们研究了随机凸优化(SCO)背景下记忆与学习之间的相互作用。我们通过学习算法关于其训练数据点所揭示的信息来定义记忆,并利用Steinke和Zakynthinou(2020)提出的条件互信息(CMI)框架来量化该信息。我们的主要结果是对学习算法的准确度与其CMI之间权衡的精确刻画,回答了Livni(2023)提出的一个开放问题。我们证明,在$L^2$ Lipschitz有界设置下以及强凸条件下,每个具有超额误差$\u03b5$的学习者的CMI分别有下界$\u03a9(1/\u03b5^2)$和$\u03a9(1/\u03b5)$。我们进一步通过设计一个对手,能够在特定的SCO问题中准确识别相当一部分训练样本,从而展示了记忆在SCO学习问题中的关键作用。最后,我们列举了我们的结果的若干含义,例如基于CMI的泛化界限的局限性以及SCO问题中样本的不可压缩性。

英文摘要:

In this work, we investigate the interplay between memorization and learning in the context of \emph{stochastic convex optimization} (SCO). We define memorization via the information a learning algorithm reveals about its training data points. We then quantify this information using the framework of conditional mutual information (CMI) proposed by Steinke and Zakynthinou (2020). Our main result is a precise characterization of the tradeoff between the accuracy of a learning algorithm and its CMI, answering an open question posed by Livni (2023). We show that, in the $L^2$ Lipschitz--bounded setting and under strong convexity, every learner with an excess error $\varepsilon$ has CMI bounded below by $Ω(1/\varepsilon^2)$ and $Ω(1/\varepsilon)$, respectively. We further demonstrate the essential role of memorization in learning problems in SCO by designing an adversary capable of accurately identifying a significant fraction of the training samples in specific SCO problems. Finally, we enumerate several implications of our results, such as a limitation of generalization bounds based on CMI and the incompressibility of samples in SCO problems.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑