带噪约束学习问题的比较
Comparing Corrupted Constrained Learning Problems
中文总结 AI 辅助
本文针对约束学习场景，通过反例证明经典数据处理不等式不成立，提出广义数据处理不等式并推导其成立的充分条件。
中文摘要 AI 辅助
统计学中的一个关键结果是数据处理不等式，最初由Blackwell（1951）证明，后由DeGroot（1962）从统计不确定性角度完善。该不等式指出，无论选择何种损失函数或先验，通过随机修改另一统计实验得到的统计实验的贝叶斯风险，都不会低于原实验的贝叶斯风险。在机器学习中，该结果是信息瓶颈原理和部分特征学习技术等应用的基础。然而，机器学习问题属于约束学习问题：所用模型类不包含所有可测函数。本文给出一个简单反例，表明经典数据处理不等式在该场景下不成立。因此，我们提出广义数据处理不等式，要求联合分布的约束贝叶斯风险（针对损失函数和约束假设类）为随机修改分布上的约束贝叶斯风险的下界，且与分布选择无关。我们证明该不等式等价于由损失函数和模型类诱导的特定函数集（称为超预测集）上的集合包含条件，最终推导了该包含关系的充分条件。
英文摘要
A key result in statistics is the data processing inequality, originally proved by Blackwell (1951) and later refined by DeGroot (1962) in terms of statistical uncertainty. The latter statement claims that the Bayes risk achieved on raw data always undercuts the Bayes risk on a processed version of the same data. This seemingly contradicts empirical findings in machine learning: pre-training, representation learning, feature learning, data augmentation are techniques used to improve performance of machine learning models. We reconcile both worlds by simply accepting that machine learning problems are constrained learning problems: the model class used does not include all measurable functions. We present counterexamples showing that the classical data processing inequality fails to hold in such a setting. Hence, we formulate a generalized data processing inequality, requiring the constrained Bayes risk of a joint distribution (with respect to a loss function and a constrained model class) to lower bound the constrained Bayes risk on the stochastically modified data distribution, regardless of the choice of distribution. We show this inequality to be equivalent to a set containment condition on a specific function set induced by the loss and model class, called the superprediction set. Finally, we exploit our characterization, derive sufficient conditions for this containment and quantify the inequality-gap.
