发表机构
Department of Statistics University of Chicago; Department of Computing and Mathematical Sciences Department of Electrical Engineering California Institute of Technology（统计学系芝加哥大学; 计算与数学科学系电气工程系加州理工学院）
机构由 AI 辅助整理，请以论文原文为准。AI 中文总结
研究不同大小输入的机器学习模型泛化及评估问题，提出用随机采样映射比较不同大小输入的统一方法，刻画各采样适用领域，为相关函数类给出泛化和草图绘制率
AI 中文摘要
许多机器学习模型适用于不同大小的输入，如不同点数的点云、不同长度的令牌序列和不同节点数的图。这些模型在有限大小的有限示例上训练，那么它们从小尺寸输入到训练中未见过的大尺寸输入的泛化能力如何？此外，在大输入上评估此类模型通常成本高昂。我们提出一种统一方法，通过使用随机采样映射来比较不同大小的输入，以解决这些问题。我们考虑的采样映射是有放回采样、随机装箱和物种采样的推广。我们根据域中不同大小问题实例之间的对称性和关系，刻画了每种采样适用的应用领域。我们的框架为关于所选采样概念连续的函数类产生了明确的泛化和草图绘制率，涵盖了在不同大小的序列、图和张量上定义的大量函数族。具体例子包括测度上的矩多项式、同态密度和图的数量、置换不变变压器和图神经网络
英文摘要
Many machine learning models are defined for inputs of different sizes, such as point clouds containing different numbers of points, sequences of tokens of different lengths, and graphs on different numbers of nodes. Such models are trained on finitely many examples of necessarily limited sizes. How well do these models generalize from inputs of small size to larger inputs of size not seen during training? Furthermore, evaluating such models on large inputs is often expensive. How can we sketch large inputs to obtain smaller ones on which the model takes similar values? At the heart of both questions is the need to compare inputs of different sizes and to approximate large inputs by small ones. We present a unified approach to address these questions by using random sampling maps to compare inputs of different sizes. The sampling maps we consider are generalizations of sampling with replacement, random binning, and species sampling. We characterize the application domains in which each type of sampling is appropriate in terms of the symmetries and relations between problem instances of different sizes in the domain. Our framework yields explicit generalization and sketching rates for function classes continuous with respect to a chosen notion of sampling, encompassing large families of functions defined on sequences, graphs, and tensors of different sizes. Specific examples include moment polynomials on measures, homomorphism densities and numbers of graphs, permutation-invariant transformers, and graph neural networks.
CommentsImproved sketching rates for transformers and generalization rates for polynomials
