Quality Estimation (QE) plays an essential role in applications of Machine Translation (MT). Traditionally, a QE system accepts the original source text and translation from a black-box MT system as input. Recently, a few studies indicate that as a by-product of translation, QE benefits from the model and training data's information of the MT system where the translations come from, and it is called the "glass-box QE". In this paper, we extend the definition of "glass-box QE" generally to uncertainty quantification with both "black-box" and "glass-box" approaches and design several features deduced from them to blaze a new trial in improving QE's performance. We propose a framework to fuse the feature engineering of uncertainty quantification into a pre-trained cross-lingual language model to predict the translation quality. Experiment results show that our method achieves state-of-the-art performances on the datasets of WMT 2020 QE shared task.
Self-Evaluation of Large Language Model based on Glass-box Features
Faculty of Computing, Harbin Institute of Technology（哈尔滨工业大学计算机学院）；Baidu Inc.（百度公司）；Shandong Computer Science Center, Qilu University of Technology（齐鲁工业大学山东省计算中心）
SeSE: Black-Box Uncertainty Quantification for Large Language Models Based on Structural Information Theory
School of Cyber Science and Technology Beihang University（北航信息科学与技术学院）；School of Computer Science and Engineering Beihang University（北航计算机科学与工程学院）；Didi Chuxing（滴滴出行）；Laboratory for Big Data and Decision National University of Defense Technology（国防科技大学大数据与决策实验室）；Department of Computer Science University of Illinois Chicago（伊利诺伊大学芝加哥分校计算机科学系）
Generating with Confidence: Uncertainty Quantification for Black-box Large Language Models
University of Illinois at Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）；Carle’s Illinois College of Medicine, University of Illinois at Urbana-Champaign（卡尔伊利诺伊医学院，伊利诺伊大学厄巴纳-香槟分校）
Target leakage, not model class, explains reported accuracy in survey-based cardiovascular screening: a leakage-tiered audit of glass-box and tabular foundation models
XU Exponential University of Applied Sciences（XU指数应用科学大学）；German University of Digital Science（德国数字科学大学）；Abu Dhabi University（阿布扎比大学）；Zayed University（扎耶德大学）；Cleveland Clinic Hospital（克利夫兰诊所医院）
Estimating the Black-box LLM Uncertainty with Distribution-Aligned Adversarial Distillation
School of Computer Science and Technology, Tianjin University, China（天津大学计算机科学与技术学院）；School of Future Technology, Tianjin University, China（天津大学未来技术学院）；Georgia Tech Shenzhen Institute, Tianjin University, China（Georgia Tech深圳研究院）；School of Artificial Intelligence, Tianjin University, China（天津大学人工智能学院）
