针对MOS预测模型在多样样本上性能不稳定的问题,提出一种不确定性感知的MOS预测系统,通过异方差回归和Monte Carlo dropout分别建模偶然与认知不确定性,实现选择性预测和域外检测,显著提升了系统在真实开放环境中的实用性。
摘要:
均方意见分数(MOS)预测在特定领域取得了显著进展。然而,MOS预测模型在多样样本上不稳定的性能,给这些系统的实际应用带来了持续的挑战。在本文中,我们指出缺乏不确定性建模是阻碍MOS预测系统应用于真实和开放世界的重要限制。我们分析了MOS预测任务中不确定性的来源,并提出建立一个不确定性感知的MOS预测系统,该系统分别通过异方差回归和Monte Carlo dropout来建模偶然不确定性和认知不确定性。实验结果表明,该系统能很好地捕捉不确定性,并能够执行选择性预测和域外检测。这些能力显著增强了MOS系统在多样真实和开放世界环境中的实用价值。
英文摘要:
Mean Opinion Score (MOS) prediction has made significant progress in specific domains. However, the unstable performance of MOS prediction models across diverse samples presents ongoing challenges in the practical application of these systems. In this paper, we point out that the absence of uncertainty modeling is a significant limitation hindering MOS prediction systems from applying to the real and open world. We analyze the sources of uncertainty in the MOS prediction task and propose to establish an uncertainty-aware MOS prediction system that models aleatory uncertainty and epistemic uncertainty by heteroscedastic regression and Monte Carlo dropout separately. The experimental results show that the system captures uncertainty well and is capable of performing selective prediction and out-of-domain detection. Such capabilities significantly enhance the practical utility of MOS systems in diverse real and open-world environments.
Hyper Evidential Deep Learning to Quantify Composite Classification Uncertainty
The University of Texas at Dallas(德克萨斯大学达拉斯分校);US Army Research Laboratory(美国陆军研究实验室);University of Oslo(奥斯陆大学);Virginia Tech(弗吉尼亚理工大学);University of the District of Columbia(哥伦比亚特区大学)
Quantifying the Uncertainty of Blindly Estimated Room Embeddings Using a Dispersion-Calibrated Score
Centre for Vision Speech and Signal Processing, University of Surrey(萨里大学视觉、语音与信号处理中心);International Audio Laboratories Erlangen(埃尔朗根国际音频实验室);Fraunhofer Institute for Integrated Circuits IIS(弗劳恩霍夫集成电路研究所)
Uncertainty Quantification for Regression: A Unified Framework based on kernel scores
LMU Munich(慕尼黑莱茵恩大学);Munich Center for Machine Learning (MCML)(慕尼黑机器学习中心);DLR-German Aerospace Center University of Tromsø(德国航空航天中心-特罗姆瑟大学);German Research Center for Artificial Intelligence (DFKI, DSA)(德国人工智能研究中心)
Epistemic Uncertainty in Conformal Scores: A Unified Approach
Department of Statistics Federal University of São Carlos(统计系巴西圣卡洛斯联邦大学);Institute of Mathematics and Computer Science University of São Paulo(数学与计算机科学研究所圣保罗大学)
Better Uncertainty Calibration via Proper Scores for Classification and Beyond
German Cancer Research Center (DKFZ)(德国癌症研究中心);German Cancer Consortium (DKTK)(德国癌症联盟);Goethe University Frankfurt(法兰克福大学);Frankfurt Cancer Institute(法兰克福癌症研究所)
Between the Layers Lies the Truth: Uncertainty Estimation in LLMs Using Intra-Layer Local Information Scores
Faculty of Data and Decision Sciences, Technion --- Israel Institute of Technology(数据与决策科学学院,技术离子研究所);Faculty of Computer Science, Technion --- Israel Institute of Technology(计算机科学学院,技术离子研究所);Faculty of Biomedical Engineering, Technion --- Israel Institute of Technology(生物医学工程学院,技术离子研究所)
The Distributional Uncertainty of the SHAP score in Explainable Machine Learning
Instituto de Ciencias de la Computación, UBA-CONICET(阿根廷计算机科学与技术研究所(布宜诺斯艾利斯大学-阿根廷国家科学与技术研究理事会联合机构));Department of Computer Science, Universidad de Buenos Aires(布宜诺斯艾利斯大学计算机科学系);Carleton Univ.(卡尔顿大学);IMFD(智利数学物理研究所);Universidad San Sebastián(圣塞瓦斯蒂安大学);Department of Computer Science, University of Sheffield(谢菲尔德大学计算机科学系);Artificial Intelligence Research Institute (IIIA-CSIC)(西班牙国家研究委员会人工智能研究所);Universidad Católica de Chile(智利天主教大学);CENIA Chile(智利国家环境技术中心)
Denoising Score-Matching for Uncertainty Quantification in Inverse Problems
CEA(法国原子能和替代能源委员会);Inria(法国国家信息与自动化研究所);AIM;CNRS(法国国家科学研究中心);Université Paris-Saclay(巴黎-萨克雷大学);Université Paris Diderot(巴黎狄德罗大学);Sorbonne Paris Cité(索邦巴黎西岱大学)