Uncertainty estimation (UE) of generative large language models (LLMs) is crucial for evaluating the reliability of generated sequences. A significant subset of UE methods utilize token probabilities to assess uncertainty, aggregating multiple token probabilities into a single UE score using a scoring function. Existing scoring functions for probability-based UE, such as length-normalized scoring and semantic contribution-based weighting, are designed to solve certain aspects of the problem but exhibit limitations, including the inability to handle biased probabilities and complex semantic dependencies between tokens. To address these issues, in this work, we propose Learnable Response Scoring (LARS) function, a novel scoring function that leverages supervised data to capture complex dependencies between tokens and probabilities, thereby producing more reliable and calibrated response scores in computing the uncertainty of LLM generations. Our comprehensive experiments across question-answering and arithmetical reasoning tasks with various datasets demonstrate that LARS significantly outperforms existing scoring functions, achieving improvements of up to 16\% AUROC score.
Are Checklists Really Useful for Automatic Evaluation of Generative Tasks?
Tohoku University(东洋大学);Research and Development Center for Large Language Models, National Institute of Informatics(大型语言模型研究与开发中心,日本信息处理学会);National Institute of Informatics(日本信息处理学会)
Using GPT-4 to Augment Unbalanced Data for Automatic Scoring
Franklin College of Arts and Sciences, University of Georgia(佐治亚大学富兰克林文理学院);Mary Frances Early College of Education, University of Georgia(佐治亚大学玛丽·弗朗西斯·厄尔利教育学院)
Rubric Spans are Label Representations: Joint LLM Encoding for Short Answer Scoring
DIPF | Leibniz Institute for Research and Information in Education(DIPF | 莱布尼茨教育与信息研究所);Centre for International Student Assessment (ZIB)(国际学生评估中心(ZIB));Goethe University Frankfurt(法兰克福歌德大学)
提出 RUSPAN 框架,将评分标准视为标签表示并联合编码问题、回答与标准级别,通过评分标准无关掩码实现零样本迁移,在六种语言的 ASAS 基准上超越基线,并在 PT-ASAG 上取得显著提升。
RRC: Unlocking Generative Reward Models in LLM Reinforcement Learning via Ranking-Based Reward Construction
School of Computer Science and Engineering, Northeastern University(东北大学计算机科学与工程学院);NiuTrans Research(NiuTrans研究院);Institute of Psychology, CAS(中国科学院心理研究所);Kunming University of Science and Technology(昆明理工大学)
Markovian ODE-guided scoring can assess the quality of offline reasoning traces in language models
Department of Electrical Engineering, Indian Institute of Technology Delhi(电子工程系,印度理工学院德里);Indian Institute of Technology Delhi(印度理工学院德里);Yardi School of Artificial Intelligence, Indian Institute of Technology Delhi(人工智能学院,印度理工学院德里)