arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2310.20673cs.LGcs.CY
收藏

平衡策略:约束稀疏模型中的差异化影响

Balancing Act: Constraining Disparate Impact in Sparse Models

Mila(米拉(机构名));DIRO - Université de Montréal(蒙特利尔大学 DIRO);McGill University(麦吉尔大学)

Meraj Hashemizadeh, Juan Ramirez, Rohan Sukumaran, Golnoosh Farnadi, Simon Lacoste-Julien, Jose Gallego-Posada

AI总结:

针对模型剪枝后稀疏模型在部分数据子群上精度下降的差异化影响问题,提出一种直接约束各子群精度变化的约束优化方法,具备可解释性且能适配大模型与数百个受保护子群的场景。

摘要:

模型剪枝是一种常用方法,可让大型深度学习模型部署在计算或存储容量受限的边缘设备上。尽管稀疏模型在整个数据集层面能达到与稠密模型相当的性能,但它们在部分数据子群上会出现大幅精度下降。现有缓解剪枝引发的这种差异化影响的方法,要么依赖间接解决问题且可解释性有限的代理指标,要么计算成本随受保护子群数量增加而剧增,扩展性差。我们提出一种约束优化方法,直接解决剪枝的差异化影响:我们的公式对每个子群的稠密模型与稀疏模型之间的精度变化加以限制。这种约束选择提供了可解释的成功标准,用于判断剪枝后的模型是否达到可接受的差异水平。实验结果表明,我们的技术可可靠地扩展到涉及大型模型和数百个受保护子群的问题。

英文摘要:

Model pruning is a popular approach to enable the deployment of large deep learning models on edge devices with restricted computational or storage capacities. Although sparse models achieve performance comparable to that of their dense counterparts at the level of the entire dataset, they exhibit high accuracy drops for some data sub-groups. Existing methods to mitigate this disparate impact induced by pruning (i) rely on surrogate metrics that address the problem indirectly and have limited interpretability; or (ii) scale poorly with the number of protected sub-groups in terms of computational cost. We propose a constrained optimization approach that directly addresses the disparate impact of pruning: our formulation bounds the accuracy change between the dense and sparse models, for each sub-group. This choice of constraints provides an interpretable success criterion to determine if a pruned model achieves acceptable disparity levels. Experimental results demonstrate that our technique scales reliably to problems involving large models and hundreds of protected sub-groups.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑