What is important about the No Free Lunch theorems?
Santa Fe Institute（圣塔菲研究所）
David H. Wolpert
备注：
15 pages, 11 of main text, to be published in "Black Box Optimization, Machine Learning and No-Free Lunch Theorems", P. Pardalos, V. Rasskazova, M.N. Vrahatis, Ed., Springer
The No Free Lunch theorems prove that under a uniform distribution over induction problems (search problems or learning problems), all induction algorithms perform equally. As I discuss in this chapter, the importance of the theorems arises by using them to analyze scenarios involving {non-uniform} distributions, and to compare different algorithms, without any assumption about the distribution over problems at all. In particular, the theorems prove that {anti}-cross-validation (choosing among a set of candidate algorithms based on which has {worst} out-of-sample behavior) performs as well as cross-validation, unless one makes an assumption -- which has never been formalized -- about how the distribution over induction problems, on the one hand, is related to the set of algorithms one is choosing among using (anti-)cross validation, on the other. In addition, they establish strong caveats concerning the significance of the many results in the literature which establish the strength of a particular algorithm without assuming a particular distribution. They also motivate a ``dictionary'' between supervised learning and improve blackbox optimization, which allows one to ``translate'' techniques from supervised learning into the domain of blackbox optimization, thereby strengthening blackbox optimization algorithms. In addition to these topics, I also briefly discuss their implications for philosophy of science.
Generalized No Free Lunch Theorem for Adversarial Robustness
Criteo（Criteo公司）
这份手稿提出了一些新的不可能的结果对抗的鲁棒性在机器学习，一个非常重要的，但很大程度上是开放的问题。我们证明了，如果以类标号为条件，数据分布满足$W_2$ Talagrand运输成本不等式（例如，如果条件分布的密度是对数凹的，则满足这个条件;是具有正Ricci曲率的紧致黎曼流形上的一致测度，一旦扰动稍微大于问题中的自然噪声水平，则任何分类器都可能以高概率被反向欺骗。我们把这个结果称为强“没有免费午餐”定理，作为最近的一些结果（Tsipras et al. 2018，Fawzi et al. 2018，etc.）。可以立即恢复为非常特殊的情况。我们的理论界模拟和真实数据（MNIST）证明。最后，我们对未来可能的研究方向进行了一些推测。
No Free Lunch Theorem for Security and Utility in Federated Learning
Hong Kong University of Science and Technology（香港科技大学）；Webank（微众银行）
在联邦学习场景中，多方从各自的数据中联合学习模型，对于选择合适的算法存在两个相互冲突的目标。一方面，在\textit{semi-honest}合作伙伴在场的情况下，私人和敏感的训练数据必须尽可能地保持安全，而另一方面，为了学习效用，必须在不同方之间交换一定量的信息。这样的挑战需要隐私保护的联邦学习解决方案，它最大限度地提高了学习模型的效用，并保持了参与方的私人数据的可证明的隐私保证。本文阐述了一个一般框架，a）从统一的信息论角度阐述了隐私损失和效用损失之间的权衡，b）描绘了使用不同保护机制（包括随机化、稀疏性和同态加密）时隐私-效用权衡的定量界限。结果表明，在一般情况下，\textit{there is no free lunch for the privacy-utility trade-off}和一个交易的隐私保护与一定程度的降级效用。本文的定量分析对设计实用的联邦学习算法具有指导意义。
