arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2512.05024stat.MEcs.AIcs.LG
收藏

通过分位数曲线进行无模型的模拟器保真度评估

Model-Free Assessment of Simulator Fidelity via Quantile Curves

Department of IEOR and Data Science Institute, Columbia University(工业工程与数据科学学院,哥伦比亚大学)

Garud Iyengar, Yu-Shiou Willy Lin, Kaizheng Wang

AI总结:

本文提出通过分位数曲线评估模拟器保真度,构建置信区间以估计潜变量参数差异,支持统计推断和风险测量,适用于多种输出空间。

摘要:

随着生成式AI模型被越来越多地用于模拟现实系统,量化'仿真到现实'差距至关重要。对于感兴趣的输入设置(称为场景),真实和模拟系统都与未观察到的潜在总体参数相关联,其差异随场景变化。关键挑战是,对于任何给定场景,这种差异无法直接观测,因为两个系统只能通过有限样本访问,且样本量在不同场景中往往异质。因此,标准预测推断方法不适用,因为它们量化可观察输出的不确定性而非潜在总体参数。为此,我们构建这些潜在参数的置信区间,并利用它们得到一个稳健的仿真到现实差异代理。然后估计此代理的分位数函数,以获得模拟器的分布级风险概况,支持广泛的统计总结,包括在新场景中对真实输出分布的统计推断、计算如条件风险价值(CVaR)等风险度量,以及在不同模拟器之间进行原则性比较。我们的方法是模型无关的,适用于一般输出空间,如分类调查响应和连续多维数据。我们通过评估四个主要LLMs在WorldValueBench数据集上与人类人口的一致性,展示了该方法的实用性。

英文摘要:

As generative AI models are increasingly used to simulate real-world systems, quantifying the ``sim-to-real'' gap is critical. For each input setting of interest -- which we call a \emph{scenario}, such as a survey question or operating condition -- the real and simulated systems are associated with unobserved latent population parameters, and their discrepancy varies across scenarios. A fundamental challenge is that, for any given scenario, this discrepancy cannot be observed directly, since both systems are accessible only through finite samples, often of heterogeneous sizes across scenarios. Standard predictive inference methods are therefore ill-suited, as they quantify uncertainty in observable outputs rather than latent population parameters. To address this, we construct confidence sets for these latent parameters and use them to derive a robust proxy for the sim-to-real discrepancy. We then estimate the quantile function of this proxy to obtain a distribution-level risk profile of the simulator, which supports a broad range of statistical summaries, including statistical inference for the real output distribution in a new scenario, the calculation of risk measures like Conditional Value-at-Risk (CVaR), and principled comparisons across simulators. Our method is model-agnostic and handles general output spaces, such as categorical survey responses and continuous multi-dimensional data. We demonstrate the practical utility of this method by evaluating the alignment of four major LLMs with human populations on the WorldValueBench dataset.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑