通过约束最大似然估计实现鲁棒的大语言模型性能认证
Robust LLM Performance Certification via Constrained Maximum Likelihood Estimation
AI总结：
本文提出基于约束最大似然估计的方法，结合人工标注集、自动标注数据和领域约束，提升大语言模型失败率估计的准确性和稳定性，优于现有方法。
摘要：
能够严格估计大型语言模型（LLM）的失效率是其安全部署的前提。然而，从业者常面临昂贵的人工金标准与可能严重偏见的自动标注方案（如
英文摘要：
The ability to rigorously estimate the failure rates of large language models (LLMs) is a prerequisite for their safe deployment. Currently, however, practitioners often face a tradeoff between expensive human gold standards and potentially severely-biased automatic annotation schemes such as "LLM-as-a-Judge" labeling. In this paper, we propose a new, practical, and efficient approach to LLM failure rate estimation based on constrained maximum-likelihood estimation (MLE). Our method integrates three distinct signal sources: (i) a small, high-quality human-labeled calibration set, (ii) a large corpus of LLM-judge annotations, and, most importantly, (iii) additional side information via domain-specific constraints derived from known bounds on judge performance statistics. We validate our approach through a comprehensive empirical study, benchmarking it against state-of-the-art baselines like Prediction-Powered Inference (PPI). Across diverse experimental regimes -- spanning varying judge accuracies, calibration set sizes, and LLM failure rates -- our constrained MLE consistently delivers more accurate and lower-variance estimates than existing methods. By moving beyond the "black-box" use of automated judges to a flexible framework, we provide a principled, interpretable, and scalable pathway towards LLM failure-rate certification.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
