Aligning Language Model Benchmarks with Pairwise Preferences
School of Data Science, University of Virginia(弗吉尼亚大学数据科学学院);Imperial College London(伦敦帝国理工学院);Thomson Reuters Foundational Research(汤姆森路透基础研究);Department of Electrical Engineering and Computer Science, UC Berkeley and UCSF(伯克利大学电气工程与计算机科学系及旧金山大学)
Marco Gutierrez, Xinyi Leng, Hannah Cyberey, Jonathan Richard Schwarz, Ahmed Alaa, Thomas Hartvigsen
Language model benchmarks are pervasive and computationally-efficient proxies for real-world downstream performance. However, many recent works find that benchmarks often fail to predict downstream utility. While some works have begun diagnosing sources of misalignment, there remain no ways to systematically update benchmarks to align their scores with downstream usage. Towards bridging this gap, we introduce and study \textit{benchmark alignment}, where we use information about downstream model performance to automatically update benchmarks, specifically aiming to update static benchmarks so they generalizably rank models according to new pairwise preferences. Our experiments involving 4576 language models and 6 benchmarks show that reweighting benchmark items can successfully rank unseen models, even generalizing across model scales in most cases. And while naive alignment unsurprisingly requires large numbers of models and benchmark questions, an oracle experiment suggests this could be reduced to as few as 20 well-chosen models. Overall, our work takes a step towards efficiently aligning benchmark development with downstream tasks.\footnote{All of our code, models, and data are publicly-available.
SimulPL: Aligning Human Preferences in Simultaneous Machine Translation
School of Artificial Intelligence, University of Chinese Academy of Sciences(中国科学院大学人工智能学院);Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所);Graduate School of Translation and Interpretation, Beijing Foreign Studies University(北京外国语大学高级翻译学院)
TPO: Aligning Large Language Models with Multi-branch & Multi-step Preference Trees
School of Computer Science, Peking University(北京大学计算机科学系);Center on Frontiers of Computing Studies, Peking University(北京大学前沿计算研究中心);National Research and Engineering Center of Software Engineering, Peking University(北京大学软件工程研究中心)
Aligning LLMs through Multi-perspective User Preference Ranking-based Feedback for Programming Question Answering
School of Computer Science and Technology, University of Science and Technology of China(中国科学技术大学计算机科学与技术学院);Hefei University of Technology(合肥工业大学);School of Artificial Intelligence and Data Science, University of Science and Technology of China(中国科学技术大学人工智能与数据科学学院);University of Toronto(多伦多大学)
Synergistic Weak-Strong Collaboration by Aligning Preferences
Microsoft Research(微软研究院);University of Illinois Urbana-Champaign(伊利诺伊大学厄巴纳-香槟分校);University of North Carolina at Chapel Hill(北卡罗来纳大学教堂山分校);Nanyang Technological University(南洋理工大学)
Aligning Large Language Models from Self-Reference AI Feedback with one General Principle
School of Computer Science, Fudan University(复旦大学计算机科学学院);Shanghai Artificial Intelligence Laboratory(上海人工智能实验室);The University of Sydney(悉尼大学);Nanyang Technological University(南洋理工大学)
提出自参考 AI 反馈框架,让 Llama2-Chat 依据通用原则、自我批评与自一致性生成高质量偏好数据,显著提升强化学习对齐效果。
SIPO: Stabilized and Improved Preference Optimization for Aligning Diffusion Models
Shanghai Science and Intelligence Institute, Shanghai, China(上海科学与智能研究所);Fudan University, Shanghai, China(复旦大学);Australian Institute for Machine Learning, The University of Adelaide(澳大利亚机器学习研究所,阿德莱德大学)
Human Preference Score: Better Aligning Text-to-Image Models with Human Preference
The Chinese University of Hong Kong(香港中文大学);SenseTime Research(商汤科技研究院);Qing Yuan Research Institute, Shanghai Jiao Tong University(上海交通大学清源研究院);Centre for Perceptual and Interactive Intelligence (CPII)(感知与交互智能中心(CPII));Shanghai AI Laboratory(上海人工智能实验室)
Reinforcement Learning with Pairwise Preferences in Long-Term Decision Problems
School of Computer Science, McGill University, Montreal, Quebec, Canada(麦吉尔大学计算机科学学院);Mila - Quebec AI Institute, Montreal, Quebec, Canada(魁北克人工智能研究所);Department of Electrical Engineering, Stanford University, Stanford, California, USA(斯坦福大学电气工程系)
Aligning Human Sense: Calibrated Distributional Reward Learning for Video Generation
City University of Hong Kong(香港城市大学);ShanghaiTech University(上海科技大学);University of California, Berkeley(加利福尼亚大学伯克利分校);The Ohio State University(俄亥俄州立大学);University of Glasgow(格拉斯哥大学);Columbia University(哥伦比亚大学);University of Arizona(亚利桑那大学);GMI Cloud;National University of Singapore(新加坡国立大学);DeciLix Lab
Directly Aligning the Full Diffusion Trajectory with Fine-Grained Human Preference
School of Science and Engineering, The Chinese University of Hong Kong, Shenzhen(香港中文大学(深圳)科学与工程学院);Shenzhen International Graduate School, Tsinghua University(清华大学深圳国际研究生院)