arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2603.23055stat.MLcs.ITcs.LGmath.IT
收藏

后选择分布模型评估

Post-Selection Distributional Model Evaluation

Institute for Intelligent Networked Systems (INSI)(智能网络系统研究所)

Amirmohammad Farzaneh, Osvaldo Simeone

AI总结:

本文提出PS-DME框架,用于在数据依赖的模型预选后进行统计有效的分布模型评估,通过控制后选择虚无覆盖率提升样本效率,实验证明其在性能-可靠性权衡中可靠性。

摘要:

正式的模型评估方法通常证明模型满足规定的性能指标(KPI)水平。然而,在许多应用中,相关的目标KPI水平可能无法预先确定,用户可能希望通过分析测试时性能与可靠性之间的完整权衡来比较候选模型。这项任务要求可靠估计测试时KPI分布,这变得更加复杂,因为同一数据通常用于预选候选模型集和估计其KPI分布,导致潜在的后选择偏差。本文介绍了后选择分布模型评估(PS-DME),一种通用的框架,用于在任意数据依赖的模型预选后进行统计有效的分布模型评估。基于e值,PS-DME控制后选择虚无覆盖率(FCR)以确保分布KPI估计的统计有效性,并建立了显式条件,证明其比基于样本分割的基线方法在样本效率上更具优势。在合成数据、文本到SQL解码与大语言模型以及电信网络性能评估中的实验表明,PS-DME能够在各种可靠性水平上可靠地比较候选配置,支持对性能-可靠性权衡的统计可靠探索。

英文摘要:

Formal model evaluation methods typically certify that a model satisfies a prescribed target key performance indicator (KPI) level. However, in many applications, the relevant target KPI level may not be known a priori, and the user may instead wish to compare candidate models by analyzing the full trade-offs between performance and reliability achievable at test time by the models. This task, requiring the reliable estimate of the test-time KPI distributions, is made more complicated by the fact that the same data must often be used both to pre-select a subset of candidate models and to estimate their KPI distributions, causing a potential post-selection bias. In this work, we introduce post-selection distributional model evaluation (PS-DME), a general framework for statistically valid distributional model assessment after arbitrary data-dependent model pre-selection. Building on e-values, PS-DME controls post-selection false coverage rate (FCR) for the distributional KPI estimates and we establish explicit conditions under which it is provably more sample efficient than a baseline method based on sample splitting. Experiments on synthetic data, text-to-SQL decoding with large language models, and telecom network performance evaluation demonstrate that PS-DME enables reliable comparison of candidate configurations across a range of reliability levels, supporting the statistically reliable exploration of performance--reliability trade-offs.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑