Automated assessment in natural language generation is a challenging task. Instruction-tuned large language models (LLMs) have shown promise in reference-free evaluation, particularly through comparative assessment. However, the quadratic computational complexity of pairwise comparisons limits its scalability. To address this, efficient comparative assessment has been explored by applying comparative strategies on zero-shot LLM probabilities. We propose a framework for finetuning LLMs for comparative assessment to align the model's output with the target distribution of comparative probabilities. By training on soft probabilities, our approach improves state-of-the-art performance while maintaining high performance with an efficient subset of comparisons.
Can Large Language Models Outperform Non-Experts in Poetry Evaluation? A Comparative Study Using the Consensual Assessment Technique
School of Computing, University of Kent（肯特大学计算学院）；Cheriton School of Computer Science, University of Waterloo（滑铁卢大学切里顿计算机科学学院）；Computing and Mathematical Sciences Department, University of Leicester（莱斯特大学计算与数学科学系）
To Compare, or Not to Compare: On Methodological Practices in Evaluating Social Bias
Tsinghua University（清华大学）；The Hebrew University of Jerusalem（耶路撒冷希伯来大学）；Ubiquitous Knowledge Processing Lab (UKP Lab), Department of Computer Science, TU Darmstadt and National Research Center for Applied Cybersecurity ATHENE（普适知识处理实验室（UKP Lab），计算机科学系，达姆施塔特工业大学及国家应用网络安全研究中心ATHENE）
Systematic Exploration of Dialogue Summarization Approaches for Reproducibility, Comparative Assessment, and Methodological Innovations for Advancing Natural Language Processing in Abstractive Summarization
SRM Institute of Science and Technology（SRM科学技术学院）；SRM University, AP（SRM大学（安得拉邦））
Semantic relatedness in DBpedia: A comparative and experimental assessment
Istituto di Analisi dei Sistemi ed Informatica “Antonio Ruberti”（“安东尼奥·鲁贝蒂”系统分析与计算机科学研究所）；Consiglio Nazionale delle Ricerche (IASI-CNR)（意大利国家研究委员会（IASI-CNR））
wav2vec and its current potential to Automatic Speech Recognition in German for the usage in Digital History: A comparative assessment of available ASR-technologies for the use in cultural heritage contexts
Conversation AI Dialog for Medicare powered by Finetuning and Retrieval Augmented Generation
University of Florida（佛罗里达大学）；National Institute of Technology Calicut（卡利卡特国家理工学院）；SRM, Chennai（SRM大学（金奈））；University of Cincinnati（辛辛那提大学）；Dartmouth College（达特茅斯学院）；Arizona State University（亚利桑那州立大学）
