Auto-evaluation is crucial for assessing response quality and offering feedback for model development. Recent studies have explored training large language models (LLMs) as generative judges to evaluate and critique other models' outputs. In this work, we investigate the idea of learning from both positive and negative data with preference optimization to enhance the evaluation capabilities of LLM judges across an array of different use cases. We achieve this by employing three approaches to collect the preference pairs for different use cases, each aimed at improving our generative judge from a different perspective. Our comprehensive study over a wide range of benchmarks demonstrates the effectiveness of our method. In particular, our generative judge achieves the best performance on 10 out of 13 benchmarks, outperforming strong baselines like GPT-4o and specialized judge models. Further analysis show that our judge model robustly counters inherent biases such as position and length bias, flexibly adapts to any evaluation protocol specified by practitioners, and provides helpful language feedback for improving downstream generator models.
Meta AI；MIT（麻省理工学院）；Johns Hopkins（约翰斯·霍普金斯大学）；Purdue University（普渡大学）
Targeted syntactic evaluations of language models ask whether models show stable preferences for syntactically acceptable content over minimal-pair unacceptable inputs. Most targeted syntactic evaluation datasets ask models to make these judgements with just a single context-free sentence as input. This does not match language models' training regime, in which input sentences are always highly contextualized by the surrounding corpus. This mismatch raises an important question: how robust are models' syntactic judgements in different contexts? In this paper, we investigate the stability of lan…
Legal Prompt Engineering (LPE) or Legal Prompting is a process to guide and assist a large language model (LLM) with performing a natural legal language processing (NLLP) skill. Our goal is to use LPE with LLMs over long legal documents for the Legal Judgement Prediction (LJP) task. We investigate the performance of zero-shot LPE for given facts in case-texts from the European Court of Human Rights (in English) and the Federal Supreme Court of Switzerland (in German, French and Italian). Our results show that zero-shot LPE is better compared to the baselines, but it still falls short compared …
DGPO: Beyond Pairwise Preferences with Directional Consistent Groupwise Optimization
Information Hub, The Hong Kong University of Science and Technology (Guangzhou), China（香港科学与技术大学（广州）信息中心，中国）；The Hong Kong University of Science and Technology, Hong Kong SAR（香港科学与技术大学，香港特别行政区）
Cal-DPO: Calibrated Direct Preference Optimization for Language Model Alignment
Artificial Intelligence Research Laboratory, Pennsylvania State University（宾夕法尼亚州立大学人工智能研究实验室）；University of Chinese Academy of Sciences（中国科学院大学）；Tencent AI Lab（腾讯人工智能实验室）
Training-Free Cultural Alignment of Large Language Models via Persona Disagreement
Faculty of Information and Technology, University of Science, Vietnam National University（信息与技术学院，科学大学，越南国家大学）；Department of Computer Science, University of Warwick（计算机科学系，沃里克大学）；School of Computing, Engineering and Digital Technologies, Teesside University（计算、工程和数字技术学院，泰赛德大学）
FedPDPO: Federated Personalized Direct Preference Optimization for Large Language Model Alignment
College of Intelligence and Computing, Tianjin University, Tianjin, China（智能与计算学院，天津大学，天津，中国）；College of Computing and Data Science, Nanyang Technological University, Singapore（计算与数据科学学院，南洋理工大学，新加坡）
Beyond Uniform Forgetting: A Study of Sequential Direct Preference Optimization Across Preference Settings
Network Analysis and Social Influence Modelling (NASIM) Lab（网络分析与社会影响建模实验室）；School of Physics Maths and Computing, The University of Western Australia（西澳大学物理数学与计算学院）；School of Psychological Science, The University of Western Australia（西澳大学心理科学学院）；School of Computing, Macquarie University（麦考瑞大学计算机学院）
We present COMET, a neural framework for training multilingual machine translation evaluation models which obtains new state-of-the-art levels of correlation with human judgements. Our framework leverages recent breakthroughs in cross-lingual pretrained language modeling resulting in highly multilingual and adaptable MT evaluation models that exploit information from both the source input and a target-language reference translation in order to more accurately predict MT quality. To showcase our framework, we train three models with different types of human judgements: Direct Assessments, Human…
Both Direct and Indirect Evidence Contribute to Dative Alternation Preferences in Language Models
Department of Linguistics, The University of Texas at Austin（德克萨斯大学奥斯汀分校语言学系）；Department of Linguistics and Philology, Uppsala University（乌普萨拉大学语言学与哲学系）
