As generative AI models are increasingly used to simulate real-world systems, quantifying the ``sim-to-real'' gap is critical. For each input setting of interest -- which we call a \emph{scenario}, such as a survey question or operating condition -- the real and simulated systems are associated with unobserved latent population parameters, and their discrepancy varies across scenarios. A fundamental challenge is that, for any given scenario, this discrepancy cannot be observed directly, since both systems are accessible only through finite samples, often of heterogeneous sizes across scenarios. Standard predictive inference methods are therefore ill-suited, as they quantify uncertainty in observable outputs rather than latent population parameters. To address this, we construct confidence sets for these latent parameters and use them to derive a robust proxy for the sim-to-real discrepancy. We then estimate the quantile function of this proxy to obtain a distribution-level risk profile of the simulator, which supports a broad range of statistical summaries, including statistical inference for the real output distribution in a new scenario, the calculation of risk measures like Conditional Value-at-Risk (CVaR), and principled comparisons across simulators. Our method is model-agnostic and handles general output spaces, such as categorical survey responses and continuous multi-dimensional data. We demonstrate the practical utility of this method by evaluating the alignment of four major LLMs with human populations on the WorldValueBench dataset.
Learning to Simulate: Generative Metamodeling via Quantile Regression
Department of Industrial and Systems Engineering, University of Minnesota(明尼苏达大学工业与系统工程系);School of Data Science, Fudan University(复旦大学数据科学学院);School of Management, Fudan University(复旦大学管理学院);Department of Decision Analytics and Operations, City University of Hong Kong(香港城市大学决策分析与运营系);Department of Industrial Engineering and Decision Analytics, The Hong Kong University of Science and Technology(香港科技大学工业工程与决策分析系)
Statistical Model Checking Beyond Means: Quantiles, CVaR, and the DKW Inequality (extended version)
Technical University of Denmark(丹麦技术大学);University of Twente(特文特大学);Lancaster University Leipzig(兰开斯特大学莱比锡分校);Ruhr University Bochum(波鸿鲁尔大学);Dresden University of Technology(德累斯顿工业大学)
MF-GLaM: A multifidelity stochastic emulator using generalized lambda models
Chair of Risk, Safety and Uncertainty Quantification, ETH Zürich(苏黎世联邦理工学院风险、安全与不确定性量化教席);Laboratory of Signals and Systems, CentraleSupélec, Université Paris-Saclay(巴黎萨克雷大学中央理工-高等电力学院信号与系统实验室)