For evaluating generation systems, automatic metrics such as BLEU cost nothing to run but have been shown to correlate poorly with human judgment, leading to systematic bias against certain model improvements. On the other hand, averaging human judgments, the unbiased gold standard, is often too expensive. In this paper, we use control variates to combine automatic metrics with human evaluation to obtain an unbiased estimator with lower cost than human evaluation alone. In practice, however, we obtain only a 7-13% cost reduction on evaluating summarization and open-response question answering systems. We then prove that our estimator is optimal: there is no unbiased estimator with lower cost. Our theory further highlights the two fundamental bottlenecks---the automatic metric and the prompt shown to human evaluators---both of which need to be improved to obtain greater cost savings.
A Heuristic Perspective on Debiasing Language Models
Inner Mongolia University（内蒙古大学）；Xiamen University（厦门大学）；University of Sydney（悉尼大学）；Nanjing University of Science and Technology（南京理工大学）；Harbin Institute of Technology（哈尔滨工业大学）；Zhejiang University（浙江大学）
Information Gain-Guided Causal Intervention for Autonomous Debiasing Large Language Models
Research Center for Social Computing and Interactive Robotics（社会计算与交互机器人研究中心）；Harbin Institute of Technology（哈尔滨工业大学）；Beijing Academy of Artificial Intelligence（北京人工智能研究院）
Debiasing Multimodal Large Language Models via Noise-Aware Preference Optimization
Institute of Information Engineering, Chinese Academy of Sciences（中国科学院信息工程研究所）；School of Cyber Security, University of Chinese Academy of Sciences（中国科学院大学网络空间安全学院）；Baidu Inc.（百度公司）
CogTree: Cognition Tree Loss for Unbiased Scene Graph Generation
Institute of Information Engineering, Chinese Academy of Sciences（中国科学院信息工程研究所）；Peking University（北京大学）；University of Adelaide（阿德莱德大学）；Beihang University（北京航空航天大学）
