多样化集成:众包机器学习的一项实验
Diversified Ensembling: An Experiment in Crowdsourced Machine Learning
AI总结:
本文提出了一个多样化众包机器学习框架,允许参与者基于公平性或专业知识进行专门化,并通过46个团队预测收入的实验评估了该框架,给出了系统架构与部署指导。
摘要:
在Kaggle等竞赛平台上进行众包机器学习是生成准确模型的一种流行且通常有效的方法。通常,各团队通过在留出集上的整体误差来竞争最准确的模型,在这类竞赛接近尾声时,排行榜前列的团队通常会在平台机制之外集成或平均他们的模型,以获得最终的最佳全局模型。在arXiv:2201.10408中,作者在公平机器学习的背景下开发了一种替代性的众包框架,以便在子群体不公平性存在且可识别时,将社区反馈整合到模型中。与经典的众包机器学习不同,在那里,参与者通过致力于子问题(例如服务于公平性的人口统计子群体)来刻意专门化他们的努力。在这里,我们对这项工作采取了更广阔的视角:我们注意到在该框架内,参与者既可以为了公平性而进行专门化,也可以仅仅是为了迎合他们特定的专业知识(例如,在图像分类任务中专注于识别鸟类物种)。与传统的众包不同,这使得参与者的努力得以多样化,并可能为更广泛的个人提供参与机制(例如,对特定公平性问题有洞察力的机器学习新手)。我们提出了对该框架的首次中等规模实验评估,共有46个参与团队尝试生成模型,以从American Community Survey数据中预测收入。我们对各团队的方法进行了实证分析,并讨论了我们开发的全新系统架构。基于此,我们为如何最好地部署此类框架提供了具体指导。
英文摘要:
Crowdsourced machine learning on competition platforms such as Kaggle is a popular and often effective method for generating accurate models. Typically, teams vie for the most accurate model, as measured by overall error on a holdout set, and it is common towards the end of such competitions for teams at the top of the leaderboard to ensemble or average their models outside the platform mechanism to get the final, best global model. In arXiv:2201.10408, the authors developed an alternative crowdsourcing framework in the context of fair machine learning, in order to integrate community feedback into models when subgroup unfairness is present and identifiable. There, unlike in classical crowdsourced ML, participants deliberately specialize their efforts by working on subproblems, such as demographic subgroups in the service of fairness. Here, we take a broader perspective on this work: we note that within this framework, participants may both specialize in the service of fairness and simply to cater to their particular expertise (e.g., focusing on identifying bird species in an image classification task). Unlike traditional crowdsourcing, this allows for the diversification of participants' efforts and may provide a participation mechanism to a larger range of individuals (e.g. a machine learning novice who has insight into a specific fairness concern). We present the first medium-scale experimental evaluation of this framework, with 46 participating teams attempting to generate models to predict income from American Community Survey data. We provide an empirical analysis of teams' approaches, and discuss the novel system architecture we developed. From here, we give concrete guidance for how best to deploy such a framework.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。