梯子：机器学习竞赛的可靠排行榜
The Ladder: A Reliable Leaderboard for Machine Learning Competitions
摘要：
机器学习竞赛的组织者面临的问题是保持准确的排行榜，该排行榜忠实地代表每个参赛团队的最佳提交的质量。使这个估计问题特别具有挑战性的是它的顺序和自适应性质。由于参与者被允许反复评估他们在排行榜上的提交，他们可能会开始过度拟合支持排行榜的保持数据。很少有理论结果给出了如何设计一个可靠的排行榜可操作的建议。因此，现有的方法往往诉诸于知之甚少的算法，如限制答案的位精度和重新提交的速度。在这项工作中，我们引入了一个概念的“排行榜的准确性”的竞争的格式。我们介绍了一个自然的算法，称为“梯子”，并证明它同时支持强有力的理论保证，在一个完全自适应的估计模型，承受实际的对抗攻击，并实现高实用性的真正提交文件从Kaggle主办的实际竞争。值得注意的是，我们能够回避最近一个强大的自适应风险估计的硬度结果，该结果在看似非常相似的准确性概念下排除了像我们这样的算法。在实际应用中，我们提供了一个完全无参数的算法变体，可以部署在真正的比赛中，不需要任何调整。
英文摘要：
The organizer of a machine learning competition faces the problem of maintaining an accurate leaderboard that faithfully represents the quality of the best submission of each competing team. What makes this estimation problem particularly challenging is its sequential and adaptive nature. As participants are allowed to repeatedly evaluate their submissions on the leaderboard, they may begin to overfit to the holdout data that supports the leaderboard. Few theoretical results give actionable advice on how to design a reliable leaderboard. Existing approaches therefore often resort to poorly understood heuristics such as limiting the bit precision of answers and the rate of re-submission. In this work, we introduce a notion of "leaderboard accuracy" tailored to the format of a competition. We introduce a natural algorithm called "the Ladder" and demonstrate that it simultaneously supports strong theoretical guarantees in a fully adaptive model of estimation, withstands practical adversarial attacks, and achieves high utility on real submission files from an actual competition hosted by Kaggle. Notably, we are able to sidestep a powerful recent hardness result for adaptive risk estimation that rules out algorithms such as ours under a seemingly very similar notion of accuracy. On a practical note, we provide a completely parameter-free variant of our algorithm that can be deployed in a real competition with no tuning required whatsoever.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
