arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2409.02347cs.LG
收藏

通过成分选择与多维缩放理解功能多样性在权重集成中的作用

Understanding the Role of Functional Diversity in Weight-Ensembling with Ingredient Selection and Multidimensional Scaling

Harvard University(哈佛大学);Microsoft Research(微软研究院)

Alex Rojas, David Alvarez-Melis

AI总结:

本文提出两种新的权重集成方法,结合可视化工具研究功能多样性对模型性能的影响,发现高多样性可增强权重集成,但仅靠多样性提升准确率有限,且位置不同的模型采样同样有助于改进。

摘要:

权重集成是指将多个神经网络的参数直接平均为单一模型。它们已在分布内(ID)和分布外(OOD)展现出泛化能力,但这一能力尚未被完全理解,尽管人们认为它们成功利用了每个不同模型所赋予的功能多样性。给定一组模型,目前也不清楚哪种组合能产生最优的权重集成;当前最优方法是一种线性时间的“贪心”方法。我们引入两种新颖的权重集成方法,以研究性能动态与每种方法决定如何使用功能多样性成分的方式之间的联系,这类似于预测集成文献中的多样性鼓励机制。我们开发了一个可视化工具,用于解释每种算法如何探索由成对距离定义的各个域,以进一步研究选择过程和算法的收敛性。实证分析提供了新的视角,进一步强化了高多样性如何增强权重集成,同时限定了仅靠多样性提高准确率的程度。我们还证明,采样位置不同的模型也能对权重集成的改进做出同样有意义的贡献。

英文摘要:

Weight-ensembles are formed when the parameters of multiple neural networks are directly averaged into a single model. They have demonstrated generalization capability in-distribution (ID) and out-of-distribution (OOD) which is not completely understood, though they are thought to successfully exploit functional diversity allotted by each distinct model. Given a collection of models, it is also unclear which combination leads to the optimal weight-ensemble; the SOTA is a linear-time ``greedy" method. We introduce two novel weight-ensembling approaches to study the link between performance dynamics and the nature of how each method decides to use apply the functionally diverse components, akin to diversity-encouragement in the prediction-ensemble literature. We develop a visualization tool to explain how each algorithm explores various domains defined via pairwise-distances to further investigate selection and algorithms' convergence. Empirical analyses shed perspectives which reinforce how high-diversity enhances weight-ensembling while qualifying the extent to which diversity alone improves accuracy. We also demonstrate that sampling positionally distinct models can contribute just as meaningfully to improvements in a weight-ensemble.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑