Given a black-box AI system and a task, at what confidence level can a practitioner trust the system's output? We answer with a reliability level -- a single number per system-task pair, derived from self-consistency sampling and conformal calibration, that serves as a black-box deployment gate with exact, finite-sample, distribution-free guarantees. Self-consistency sampling reduces uncertainty exponentially; conformal calibration guarantees correctness within 1/(n+1) of the target level, regardless of the system's errors -- made transparently visible through larger answer sets for harder questions. Weaker models earn lower reliability levels (not accuracy -- see Definition 2.4): GPT-4.1 earns 94.6% on GSM8K and 96.8% on TruthfulQA, while GPT-4.1-nano earns 89.8% on GSM8K and 66.5% on MMLU. We validate across five benchmarks, five models from three families, and both synthetic and real data. Conditional coverage on solvable items exceeds 0.93 across all configurations; sequential stopping reduces API costs by around 50%.
Agentic Calibration of Grey-Box Simulation Models: An LLM-Driven Alternative
Catalan Institute of Oncology-IDIBELL(加泰罗尼亚肿瘤研究所-IDIBELL);Autonomous University of Barcelona(巴塞罗那自治大学);Centro de Investigación Biomédica en Red de Epidemiología y Salud Pública(国家公共卫生与流行病学网络生物医学研究中心);Artificial Intelligence Research Institute, IIIA-CSIC(人工智能研究所,西班牙科学研究委员会-人工智能研究所)
Conformal Calibration: Ensuring the Reliability of Black-Box AI in Wireless Systems
King’s Communications, Learning and Information Processing (KCLIP) lab(国王通讯、学习与信息处理实验室);Centre for Intelligent Information Processing Systems (CIIPS)(智能信息处理系统中心);Department of Engineering(工程系)
Online Regime-aware Calibration for Black-box Social Simulators via Posterior-assisted Evolutionary Dynamic Optimization
Guangdong Provincial Key Laboratory of Brain-Inspired Intelligent Computation, Southern University of Science and Technology, Shenzhen 518055, China(脑启发智能计算广东省重点实验室,南方科技大学,深圳518055,中国);Department of Computer Science and Engineering(计算机科学与工程系);Department of Statistics and Data Science(统计与数据科学系);Zhongguancun Academy, Beijing 100094, China(中关村学院,北京100094,中国);School of Data Science, Lingnan University, Hong Kong SAR, China(数据科学学院,岭南大学,香港特别行政区,中国)
MUC: Machine Unlearning for Contrastive Learning with Black-box Evaluation
University of Waterloo(滑铁卢大学);University of Ottawa(渥太华大学);Alibaba(阿里巴巴);CISPA Helmholtz Center for Information Security(信息安全赫尔姆霍兹中心);Vector Institute(向量研究所);Academy of Mathematics and Systems Science, Chinese Academy of Sciences(中国科学院数学与系统科学研究院);University of Chinese Academy of Sciences(中国科学院大学)
CertDW: Towards Certified Dataset Ownership Verification via Conformal Calibration
School of Control and Computer Engineering, North China Electric Power University(控制与计算机工程学院,华北电力大学);College of Computing and Data Science, Nanyang Technological University(计算与数据科学学院,南洋理工大学);School of Cybersecurity, Northwestern Polytechnical University(网络安全学院,西北工业大学);Department of Computer Science, University of Maryland(计算机科学系,马里兰大学);Alibaba Group(阿里巴巴集团)