arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2407.03557cs.LGstat.ML
收藏

最坏情况分布偏移的决策聚焦评估

Decision-Focused Evaluation of Worst-Case Distribution Shift

Carnegie Mellon University(卡内基梅隆大学)

Kevin Ren, Yewon Byun, Bryan Wilder

AI总结:

本文针对预测性资源分配场景,提出基于分层模型和子模优化的决策聚焦框架,用于识别最坏情况分布偏移,并通过真实数据证明不同指标识别出的最坏偏移存在显著差异。

摘要:

分布偏移是预测模型在实际应用中的一个关键挑战,因此需要在部署前识别可能有害的偏移。现有工作通常将这些最坏情况偏移定义为对模型个体级准确率损害最大的偏移。然而,当模型被用于做出下游群体层面的决策(如稀缺资源的分配)时,个体级准确率可能无法很好地代表当前任务的实际表现。我们提出了一个新的框架,该框架采用分层模型结构,通过同时捕捉决策问题实例内部和实例之间的偏移,来识别预测性资源分配场景中的最坏情况分布偏移。由于决策依赖于分配任务中个体的联合存在,存在组合交互作用,因此该任务比标准分布偏移设置更加困难。我们证明该问题可以被重新表述为子模优化问题,从而能够高效地近似最坏情况损失。将我们的框架应用于真实数据后,我们发现经验证据表明,由一种指标识别出的最坏情况偏移往往与由其他指标识别出的最坏情况分布存在显著差异。

英文摘要:

Distribution shift is a key challenge for predictive models in practice, creating the need to identify potentially harmful shifts in advance of deployment. Existing work typically defines these worst-case shifts as ones that most degrade the individual-level accuracy of the model. However, when models are used to make a downstream population-level decision like the allocation of a scarce resource, individual-level accuracy may be a poor proxy for performance on the task at hand. We introduce a novel framework that employs a hierarchical model structure to identify worst-case distribution shifts in predictive resource allocation settings by capturing shifts both within and across instances of the decision problem. This task is more difficult than in standard distribution shift settings due to combinatorial interactions, where decisions depend on the joint presence of individuals in the allocation task. We show that the problem can be reformulated as a submodular optimization problem, enabling efficient approximations of worst-case loss. Applying our framework to real data, we find empirical evidence that worst-case shifts identified by one metric often significantly diverge from worst-case distributions identified by other metrics.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑