语言大模型 / LLM

C2-DPO: Constrained Controlled Direct Preference Optimization

Kavosh Asadi, Julien Han, Idan Pipano, Xingzi Xu, Dominique Perrault-Joncas, Shoham Sabach, Karim Bouyarmane, Mohammad Ghavamzadeh

机构 * Amazon（亚马逊公司） ； Technion（技术学院） ； Duke（杜克大学） 纠错

专题命中 后训练与偏好优化 ：preference optimization(title,abstract)；language model(abstract)；RLHF(abstract)；分类 cs.AI、cs.LG

AMoPO: Adaptive Multi-objective Preference Optimization without Reward Models and Reference Models

Qi Liu, Jingqing Ruan, Hao Li, Haodong Zhao, Desheng Wang, Jiansong Chen, Wan Guanglu, Xunliang Cai, Zhi Zheng, Tong Xu

专题命中 后训练与偏好优化 ：preference optimization(title,abstract)；large language model(abstract)；language model(abstract)；分类 cs.AI、cs.LG

Comments Accepted by ACL 2025

Is poisoning a real threat to LLM alignment? Maybe more so than you think

Pankayaraj Pathmanathan, Souradip Chakraborty, Xiangyu Liu, Yongyuan Liang, Furong Huang

机构 * University of Maryland（马里兰大学） ； Capital One（Capital One公司） 纠错

专题命中 后训练与偏好优化 ：LLM(title)；large language model(abstract)；language model(abstract)；RLHF(abstract)

Journal ref AAAI 2025

BoA: Attention-aware Post-training Quantization without Backpropagation

Junhan Kim, Ho-young Kim, Eulrang Cho, Chungman Lee, Joonyoung Kim, Yongkweon Jeon

专题命中 后训练与偏好优化 ：post-training(title,abstract)；large language model(abstract)；language model(abstract)；分类 cs.AI、cs.LG

Comments ICML 2025

Risk-aware Direct Preference Optimization under Nested Risk Measure

Lijun Zhang, Lin Li, Yajie Qi, Huizhong Song, Yaodong Yang, Jun Wang, Wei Wei

机构 * Key Laboratory of Computational Intelligence and Chinese Information Processing of Ministry of Education, School of Computer and Information Technology, Shanxi University（教育部计算智能与中文信息处理重点实验室，计算机与信息学院，山西大学） ； Institute for AI, Peking University（人工智能研究院，北京大学） ； University College London（伦敦大学学院） 纠错

专题命中 后训练与偏好优化 ：preference optimization(title,abstract)；large language model(abstract)；language model(abstract)；分类 cs.AI、cs.LG

Safe RLHF-V: Safe Reinforcement Learning from Multi-modal Human Feedback

Jiaming Ji, Xinyu Chen, Rui Pan, Conghui Zhang, Han Zhu, Jiahao Li, Donghai Hong, Boyuan Chen, Jiayi Zhou, Kaile Wang, Juntao Dai, Chi-Min Chan, Yida Tang, Sirui Han, Yike Guo, Yaodong Yang

机构 * Peking University（北京大学） ； Hong Kong University of Science and Technology（香港科技大学） 纠错

专题命中 后训练与偏好优化 ：RLHF(title,abstract)；large language model(abstract)；language model(abstract)；分类 cs.AI、cs.LG

CREAM: Consistency Regularized Self-Rewarding Language Models

Zhaoyang Wang, Weilei He, Zhiyuan Liang, Xuchao Zhang, Chetan Bansal, Ying Wei, Weitong Zhang, Huaxiu Yao

机构 * University of North Carolina at Chapel Hill（北卡罗来纳大学教堂山分校） ； Nanyang Technological University（南洋理工大学） ； National University of Singapore（国立新加坡大学） ； Microsoft Research（微软研究院） 纠错

专题命中 后训练与偏好优化 ：language model(title,abstract)；LLM(abstract)；large language model(abstract)；分类 cs.CL、cs.LG

Comments To appear at ICLR 2025

Reward Generation via Large Vision-Language Model in Offline Reinforcement Learning

Younghwan Lee, Tung M. Luu, Donghoon Lee, Chang D. Yoo

专题命中 后训练与偏好优化 ：language model(title,abstract)；RLHF(abstract)；prompting(abstract)；分类 cs.AI、cs.LG

Comments 5 pages, ICASSP 2025. First two authors are equally contributed

TangoFlux: Super Fast and Faithful Text to Audio Generation with Flow Matching and Clap-Ranked Preference Optimization

Chia-Yu Hung, Navonil Majumder, Zhifeng Kong, Ambuj Mehrish, Amir Ali Bagherzadeh, Chuan Li, Rafael Valle, Bryan Catanzaro, Soujanya Poria

专题命中 后训练与偏好优化 ：preference optimization(title,abstract)；large language model(abstract)；language model(abstract)；分类 cs.CL、cs.AI

Comments https://tangoflux.github.io/

Learning Natural Language Constraints for Safe Reinforcement Learning of Language Agents

Jaymari Chua, Chen Wang, Lina Yao

专题命中 后训练与偏好优化 ：language agent(title)；large language model(abstract)；language model(abstract)；RLHF(abstract)

Self-Rewarding Language Models

Weizhe Yuan, Richard Yuanzhe Pang, Kyunghyun Cho, Xian Li, Sainbayar Sukhbaatar, Jing Xu, Jason Weston

专题命中 后训练与偏好优化 ：language model(title,abstract)；LLM(abstract)；prompting(abstract)；分类 cs.CL、cs.AI

Comments ICML 2024

A Shared Low-Rank Adaptation Approach to Personalized RLHF

Renpu Liu, Peng Wang, Donghao Li, Cong Shen, Jing Yang

专题命中 后训练与偏好优化 ：RLHF(title,abstract)；large language model(abstract)；language model(abstract)；分类 cs.AI、cs.LG

Comments Published as a conference paper at AISTATS 2025

Mitigating Reward Over-Optimization in RLHF via Behavior-Supported Regularization

Juntao Dai, Taiye Chen, Yaodong Yang, Qian Zheng, Gang Pan

专题命中 后训练与偏好优化 ：RLHF(title,abstract)；large language model(abstract)；language model(abstract)；分类 cs.AI、cs.LG

Comments Published as a conference paper at ICLR 2025

Human Preferences for Constructive Interactions in Language Model Alignment

Yara Kyrychenko, Jon Roozenbeek, Brandon Davidson, Sander van der Linden, Ramit Debnath

专题命中 后训练与偏好优化 ：language model(title,abstract)；LLM(abstract)；large language model(abstract)；分类 cs.CL、cs.AI

Comments 1 Figure, 1 Table, 11 pages

Robust Preference Optimization through Reward Model Distillation

Adam Fisch, Jacob Eisenstein, Vicky Zayats, Alekh Agarwal, Ahmad Beirami, Chirag Nagpal, Pete Shaw, Jonathan Berant

专题命中 后训练与偏好优化 ：preference optimization(title,abstract)；language model(abstract)；post-training(abstract)；分类 cs.CL、cs.LG

Evaluating Defences against Unsafe Feedback in RLHF

Domenic Rosati, Giles Edkins, Harsh Raj, David Atanasov, Subhabrata Majumdar, Janarthanan Rajendran, Frank Rudzicz, Hassan Sajjad

专题命中 后训练与偏好优化 ：RLHF(title,abstract)；large language model(abstract)；language model(abstract)；分类 cs.CL、cs.LG

Synthesizing Post-Training Data for LLMs through Multi-Agent Simulation

Shuo Tang, Xianghe Pang, Zexi Liu, Bohan Tang, Rui Ye, Tian Jin, Xiaowen Dong, Yanfeng Wang, Siheng Chen

专题命中 后训练与偏好优化 ：post-training(title,abstract)；large language model(abstract)；language model(abstract)；分类 cs.CL、cs.AI

Ranking Unraveled: Recipes for LLM Rankings in Head-to-Head AI Combat

Roland Daynauth, Christopher Clarke, Krisztian Flautner, Lingjia Tang, Jason Mars

专题命中 后训练与偏好优化 ：LLM(title,abstract)；large language model(abstract)；language model(abstract)；分类 cs.CL、cs.AI

$f$-PO: Generalizing Preference Optimization with $f$-divergence Minimization

Jiaqi Han, Mingjian Jiang, Yuxuan Song, Stefano Ermon, Minkai Xu

专题命中 后训练与偏好优化 ：preference optimization(title,abstract)；LLM(abstract)；language model(abstract)；分类 cs.CL、cs.LG

Journal ref AISTATS 2025

Language Models Largely Exhibit Human-like Constituent Ordering Preferences

Ada Defne Tur, Gaurav Kamath, Siva Reddy

专题命中 后训练与偏好优化 ：language model(title,abstract)；LLM(abstract)；large language model(abstract)；分类 cs.CL、cs.AI

Comments NAACL 2025 Main Conference

Improving Factual Consistency of News Summarization by Contrastive Preference Optimization

Huawen Feng, Yan Fan, Xiong Liu, Ting-En Lin, Zekun Yao, Yuchuan Wu, Fei Huang, Yongbin Li, Qianli Ma

专题命中 后训练与偏好优化 ：preference optimization(title,abstract)；large language model(abstract)；language model(abstract)；分类 cs.CL、cs.AI

The Crucial Role of Samplers in Online Direct Preference Optimization

Ruizhe Shi, Runlong Zhou, Simon S. Du

专题命中 后训练与偏好优化 ：preference optimization(title,abstract)；language model(abstract)；RLHF(abstract)；分类 cs.CL、cs.LG

Comments ICLR accepted

Simultaneous Reward Distillation and Preference Learning: Get You a Language Model Who Can Do Both

Abhijnan Nath, Changsoo Jung, Ethan Seefried, Nikhil Krishnaswamy

专题命中 后训练与偏好优化 ：language model(title)；LLM(abstract)；RLHF(abstract)；preference optimization(abstract)

LiPO: Listwise Preference Optimization through Learning-to-Rank

Tianqi Liu, Zhen Qin, Junru Wu, Jiaming Shen, Misha Khalman, Rishabh Joshi, Yao Zhao, Mohammad Saleh, Simon Baumgartner, Jialu Liu, Peter J. Liu, Xuanhui Wang

专题命中 后训练与偏好优化 ：preference optimization(title,abstract)；language model(abstract)；RLHF(abstract)；分类 cs.CL、cs.LG

Comments Accepted at NAACL 2025

More RLHF, More Trust? On The Impact of Preference Alignment On Trustworthiness

Aaron J. Li, Satyapriya Krishna, Himabindu Lakkaraju

专题命中 后训练与偏好优化 ：RLHF(title,abstract)；large language model(abstract)；language model(abstract)；分类 cs.CL、cs.AI

Context-DPO: Aligning Language Models for Context-Faithfulness

Baolong Bi, Shaohan Huang, Yiwei Wang, Tianchi Yang, Zihan Zhang, Haizhen Huang, Lingrui Mei, Junfeng Fang, Zehao Li, Furu Wei, Weiwei Deng, Feng Sun, Qi Zhang, Shenghua Liu

专题命中 后训练与偏好优化 ：language model(title,abstract)；large language model(abstract)；preference optimization(abstract)；分类 cs.CL、cs.AI

Few-shot Steerable Alignment: Adapting Rewards and LLM Policies with Neural Processes

Katarzyna Kobalczyk, Claudio Fanconi, Hao Sun, Mihaela van der Schaar

专题命中 后训练与偏好优化 ：LLM(title,abstract)；large language model(abstract)；language model(abstract)；分类 cs.AI、cs.LG

Solving the Inverse Alignment Problem for Efficient RLHF

Shambhavi Krishna, Aishwarya Sahoo

专题命中 后训练与偏好优化 ：RLHF(title,abstract)；LLM(abstract)；language model(abstract)；分类 cs.CL、cs.LG

Multi-Response Preference Optimization with Augmented Ranking Dataset

Hansle Gwon, Imjin Ahn, Young-Hak Kim, Sanghyun Park, Tae Joon Jun

专题命中 后训练与偏好优化 ：preference optimization(title,abstract)；large language model(abstract)；language model(abstract)；分类 cs.CL、cs.LG

Value Imprint: A Technique for Auditing the Human Values Embedded in RLHF Datasets

Ike Obi, Rohan Pant, Srishti Shekhar Agrawal, Maham Ghazanfar, Aaron Basiletti

专题命中 后训练与偏好优化 ：RLHF(title,abstract)；LLM(abstract)；language model(abstract)；分类 cs.AI、cs.LG