Conformal Prediction (CP) allows to perform rigorous uncertainty quantification by constructing a prediction set $C(X)$ satisfying $\mathbb{P}(Y \in C(X))\geq 1-α$ for a user-chosen $α\in [0,1]$ by relying on calibration data $(X_1,Y_1),...,(X_n,Y_n)$ from $\mathbb{P}=\mathbb{P}^{X} \otimes \mathbb{P}^{Y|X}$. It is typically implicitly assumed that $\mathbb{P}^{Y|X}$ is the "true" posterior label distribution. However, in many real-world scenarios, the labels $Y_1,...,Y_n$ are obtained by aggregating expert opinions using a voting procedure, resulting in a one-hot distribution $\mathbb{P}_{vote}^{Y|X}$. For such ``voted'' labels, CP guarantees are thus w.r.t. $\mathbb{P}_{vote}=\mathbb{P}^X \otimes \mathbb{P}_{vote}^{Y|X}$ rather than the true distribution $\mathbb{P}$. In cases with unambiguous ground truth labels, the distinction between $\mathbb{P}_{vote}$ and $\mathbb{P}$ is irrelevant. However, when experts do not agree because of ambiguous labels, approximating $\mathbb{P}^{Y|X}$ with a one-hot distribution $\mathbb{P}_{vote}^{Y|X}$ ignores this uncertainty. In this paper, we propose to leverage expert opinions to approximate $\mathbb{P}^{Y|X}$ using a non-degenerate distribution $\mathbb{P}_{agg}^{Y|X}$. We develop Monte Carlo CP procedures which provide guarantees w.r.t. $\mathbb{P}_{agg}=\mathbb{P}^X \otimes \mathbb{P}_{agg}^{Y|X}$ by sampling multiple synthetic pseudo-labels from $\mathbb{P}_{agg}^{Y|X}$ for each calibration example $X_1,...,X_n$. In a case study of skin condition classification with significant disagreement among expert annotators, we show that applying CP w.r.t. $\mathbb{P}_{vote}$ under-covers expert annotations: calibrated for $72\%$ coverage, it falls short by on average $10\%$; our Monte Carlo CP closes this gap both empirically and theoretically.
Self-supervised Conformal Prediction for Uncertainty Quantification in Imaging Problems
Technical University of Denmark(丹麦技术大学);Heriot-Watt University(赫瑞瓦特大学);Maxwell Institute for Mathematical Sciences(麦克斯韦数学研究所);Université de Paris Cité(巴黎西岱大学)
Inria(法国国家信息与自动化技术研究所);Ecole Normale Supérieure(巴黎高等师范学院);PSL Research University(巴黎 sciences-et-lettres 大学);University of California, Berkeley(加州大学伯克利分校)
Institute of Informatics, LMU Munich(慕尼黑大学信息学研究所);Munich Center for Machine Learning (MCML)(慕尼黑机器学习中心);Max Planck Institute for Informatics(马克斯·普朗克信息学研究所)
Conformal Prediction Sets for Instance Segmentation
Laboratory for Information and Decision Systems, MIT(麻省理工学院信息与决策系统实验室);Department of Electrical Engineering and Computer Science, MIT(麻省理工学院电子工程与计算机科学系);Department of Mechanical Engineering, MIT(麻省理工学院机械工程系);Institute for Data, Systems, and Society, MIT(麻省理工学院数据、系统与社会研究所)
Parametric Scaling Law of Tuning Bias in Conformal Prediction
Department of Statistics and Data Science, Southern University of Science and Technology(统计与数据科学系,南方科技大学);Department of Computer and Information Science, University of Macau(计算机与信息科学系,澳门大学)
Self-Supervised Conformal Prediction with Equivariant Bootstrapping for Image Uncertainty Quantification
Mullard Space Science Laboratory, University College London(伦敦大学学院穆拉德空间科学实验室);IRFU, CEA, Université Paris-Saclay(法国原子能和替代能源委员会IRFU,巴黎萨克雷大学);Heriot-Watt University(赫瑞瓦特大学);Maxwell Institute for Mathematical Sciences(麦克斯韦数学科学研究所);The Alan Turing Institute(艾伦·图灵研究所)
Conformal Prediction for Deep Classifier via Label Ranking
Southern University of Science and Technology(南方科技大学);ShanghaiTech University(上海科技大学);Rutgers University(罗格斯大学);University of North Carolina at Chapel Hill(北卡罗来纳大学教堂山分校);Chongqing University(重庆大学)
Instituto de Telecomunicações(电信研究所);Instituto Superior Técnico, Universidade de Lisboa(里斯本大学高等技术学院);IT University of Copenhagen(哥本哈根信息技术大学);Pioneer Centre for Artificial Intelligence(先锋人工智能中心);Unbabel(Unbabel公司)
Optimal Conformal Prediction under Epistemic Uncertainty
LMU Munich(慕尼黑莱茵姆大学);CISPA(Cybersecurity Institute for Safer Applications);LMU Munich MCML(慕尼黑莱茵姆大学MCML);Ghent University(根特大学);University of Cologne(科隆大学);DFKI(德国达姆施塔特研究所)
Khoury College of Computer Sciences Northeastern University(东北大学科里学院);Network Science Institute Northeastern University(东北大学网络科学研究所);Santa Fe Institute(圣菲研究所)
HKUST(GZ)(香港科技大学(广州));University of British Columbia(不列颠哥伦比亚大学);University of Tübingen(蒂宾根大学);Harbin Institute of Technology(哈尔滨工业大学);Beijing Institute of Mathematical Sciences and Applications(北京数学科学与应用研究院);VecML Inc.(VecML公司)
针对硬标签模糊数据导致模型过度自信的问题,提出 QLL 框架与 CPU 风险估计器,并通过混合生成数据验证其泛化提升。
Exploring the Noise Robustness of Online Conformal Prediction
Department of Statistics and Data Science, Southern University of Science and Technology(统计与数据科学系,南方科技大学);Department of Computer and Information Science, University of Macau(计算机与信息科学系,澳门大学);Center for Data Science, Zhejiang University(数据科学中心,浙江大学)
Conformalized Interactive Imitation Learning: Handling Expert Shift and Intermittent Feedback
Robotics Institute, School of Computer Science, Carnegie Mellon University(机器人研究所、计算机科学学院、卡内基梅隆大学);Departments of Statistics and Machine Learning, Carnegie Mellon University(统计学与机器学习系、卡内基梅隆大学)
SAGA: A Sequence-Adaptive Generative Architecture for Multi-Horizon Probabilistic Forecasting with Adaptive Temporal Conformal Prediction
Department of Economics, Stockholm University(斯德哥尔摩大学经济系);Institute of Social Sciences, Faculty of Economics and Administrative Sciences, Süleyman Demirel University(苏莱曼·德米雷尔大学社会科学学院,经济学与行政科学学院)
Diffusion Disambiguation Models for Partial Label Learning
College of Computer Science and Technology, Qingdao University(青岛大学计算机科学与技术学院);Department of Control Science and Engineering, College of Electronics and Information Engineering, Tongji University(电子信息工程学院控制科学与工程系,同济大学);School of Electronic Information and Electrical Engineering, Shanghai Jiao Tong University(上海交通大学电子信息与电气工程学院)
Learn2Agree: Fitting with Multiple Annotators without Objective Ground Truth
Tsinghua University(清华大学);Shenzhen Institute of Artificial Intelligence Robotics for Society(深圳人工智能与机器人研究院);South China Normal University(华南师范大学);University of Cambridge(剑桥大学);University College London(伦敦大学学院)
Towards Robust Incremental Learning under Ambiguous Supervision
School of Software Technology, Zhejiang University(浙江大学软件学院);Hangzhou High-Tech Zone (Binjiang) Institute of Blockchain and Data Security(杭州高新区(滨江)区块链与数据安全研究院);College of Computer Science and Technology, Zhejiang University(浙江大学计算机科学与技术学院);School of Computer Science and Engineering, Southeast University(东南大学计算机科学与工程学院)