The manifold hypothesis presumes that high-dimensional data lies on or near a low-dimensional manifold. While the utility of encoding geometric structure has been demonstrated empirically, rigorous analysis of its impact on the learnability of neural networks is largely missing. Several recent results have established hardness results for learning feedforward and equivariant neural networks under i.i.d. Gaussian or uniform Boolean data distributions. In this paper, we investigate the hardness of learning under the manifold hypothesis. We ask which minimal assumptions on the curvature and regularity of the manifold, if any, render the learning problem efficiently learnable. We prove that learning is hard under input manifolds of bounded curvature by extending proofs of hardness in the SQ and cryptographic settings for Boolean data inputs to the geometric setting. On the other hand, we show that additional assumptions on the volume of the data manifold alleviate these fundamental limitations and guarantee learnability via a simple interpolation argument. Notable instances of this regime are manifolds which can be reliably reconstructed via manifold learning. Looking forward, we comment on and empirically explore intermediate regimes of manifolds, which have heterogeneous features commonly found in real world data.
On the Hardness of Robustness Transfer: A Perspective from Rademacher Complexity over Symmetric Difference Hypothesis Space
The Pennsylvania State University(宾夕法尼亚州立大学);Michigan State University(密歇根州立大学);Sony AI(索尼人工智能研究院)
最近的研究表明,在$\ell_\infty$攻击下的对抗鲁棒学习比标准的领域自适应更难推广到不同的领域。如何在不同领域之间传递鲁棒性是领域自适应研究的一个关键问题。为了研究对抗性鲁棒领域自适应(或鲁棒性转移)背后的根本困难,我们提出分析控制跨领域泛化的关键复杂性度量:{ \em对称差异假设空间} $\mathcal{H} Δ\mathcal{H}$上的对抗性Rademacher复杂性。对于线性模型,我们表明,这种复杂性的对抗版本总是大于非对抗版本,这揭示了对抗鲁棒领域自适应的内在硬度。我们还建立了这个复杂性措施的上限。然后,我们将它们扩展到ReLU神经网络类,通过在二进制分类设置中上界对抗Rademacher复杂度。最后,尽管鲁棒领域自适应被证明是困难的,我们确实发现鲁棒学习和标准领域自适应之间的正相关关系。我们解释\emph{how adversarial training helps domain adaptation in terms of standard risk}。我们相信,我们的研究结果启动了对抗鲁棒领域自适应的泛化理论研究,并可能揭示分布式对抗鲁棒学习的异构来源,例如,联邦学习场景。
Generative Model Inversion Through the Lens of the Manifold Hypothesis
TMLR Group, Department of Computer Science, Hong Kong Baptist University(香港 Baptist 大学计算机科学系);Sydney AI Centre, The University of Sydney(悉尼大学人工智能中心);School of Computing and Information Systems, The University of Melbourne(墨尔本大学计算与信息系);McCombs School of Business, The University of Texas at Austin(德克萨斯大学奥斯汀分校麦科姆斯商学院)
Provably Learning Diffusion Models under the Manifold Hypothesis: Collapse and Refine
RIKEN AIP & The Institute of Statistical Mathematics(日本理化学研究所AIP及统计数学研究所);University of Sydney(悉尼大学);Agency for Science, Technology and Research & The Institute of Statistical Mathematics(科技研究局及统计数学研究所);The University of Tokyo(东京大学);Nanyang Technological University(南洋理工大学);The Institute of Statistical Mathematics(统计数学研究所)
Institute for Quantitative and Computational Biosciences, University of California, Los Angeles(加州大学洛杉矶分校定量与计算生物科学研究所);Bioinformatics Interdepartmental Program, University of California, Los Angeles(加州大学洛杉矶分校生物信息学跨系项目);Department of Epidemiology, Harvard School of Public Health(哈佛大学公共卫生学院流行病学系);Department for Integrative Biology and Physiology, University of California, Los Angeles(加州大学洛杉矶分校整合生物学与生理学系)