大模型对齐与安全 - arXivDaily 专题

2410.04612 2025-04-25 cs.LG cs.AI cs.CL 85%

Regressing the Relative Future: Efficient Policy Optimization for Multi-turn RLHF

Zhaolin Gao, Wenhao Zhan, Jonathan D. Chang, Gokul Swamy, Kianté Brantley, Jason D. Lee, Wen Sun

机构 * Cornell University（康奈尔大学）； Princeton University（普林斯顿大学）； Databricks Mosaic Research（Databricks 混合研究）； Carnegie Mellon University（卡内基梅隆大学）； Harvard University（哈佛大学）

专题命中偏好对齐：RLHF(title,abstract)；DPO(abstract)；分类 cs.CL、cs.AI、cs.LG

2502.16852 2025-02-25 cs.LG cs.AI cs.CL 85%

Improving LLM General Preference Alignment via Optimistic Online Mirror Descent

Yuheng Zhang, Dian Yu, Tao Ge, Linfeng Song, Zhichen Zeng, Haitao Mi, Nan Jiang, Dong Yu

专题命中偏好对齐：alignment(title,abstract)；RLHF(abstract)；分类 cs.CL、cs.AI、cs.LG

2502.04357 2025-02-10 cs.CL cs.AI cs.LG 85%

Reusing Embeddings: Reproducible Reward Model Research in Large Language Model Alignment without GPUs

Hao Sun, Yunyi Shen, Jean-Francois Ton, Mihaela van der Schaar

专题命中偏好对齐：alignment(title,abstract)；RLHF(abstract)；分类 cs.CL、cs.AI、cs.LG

2502.04354 2025-02-10 cs.CL cs.AI cs.LG 85%

Reviving The Classics: Active Reward Modeling in Large Language Model Alignment

Yunyi Shen, Hao Sun, Jean-François Ton

专题命中偏好对齐：alignment(title,abstract)；RLHF(abstract)；分类 cs.CL、cs.AI、cs.LG

2406.07327 2025-02-10 cs.AI cs.CL cs.LG 85%

3D-Properties: Identifying Challenges in DPO and Charting a Path Forward

Yuzi Yan, Yibo Miao, Jialian Li, Yipin Zhang, Jian Xie, Zhijie Deng, Dong Yan

专题命中偏好对齐：DPO(title,abstract)；alignment(abstract)；分类 cs.CL、cs.AI、cs.LG

Journal ref ICLR 2025

2412.14093 2024-12-23 cs.AI cs.CL cs.LG 85%

Alignment faking in large language models

Ryan Greenblatt, Carson Denison, Benjamin Wright, Fabien Roger, Monte MacDiarmid, Sam Marks, Johannes Treutlein, Tim Belonax, Jack Chen, David Duvenaud, Akbir Khan, Julian Michael, Sören Mindermann, Ethan Perez, Linda Petrini, Jonathan Uesato, Jared Kaplan, Buck Shlegeris, Samuel R. Bowman, Evan Hubinger

专题命中偏好对齐：alignment(title,abstract)；harmlessness(abstract)；分类 cs.CL、cs.AI、cs.LG

2412.04903 2024-12-17 cs.CV cs.AI cs.CL cs.LG 85%

EACO: Enhancing Alignment in Multimodal LLMs via Critical Observation

Yongxin Wang, Meng Cao, Haokun Lin, Mingfei Han, Liang Ma, Jin Jiang, Yuhao Cheng, Xiaodan Liang

专题命中偏好对齐：alignment(title,abstract)；DPO(abstract)；分类 cs.CL、cs.AI、cs.LG

Comments 19 pages

2406.02900 2024-11-06 cs.LG cs.AI cs.CL 85%

Scaling Laws for Reward Model Overoptimization in Direct Alignment Algorithms

Rafael Rafailov, Yaswanth Chittepu, Ryan Park, Harshit Sikchi, Joey Hejna, Bradley Knox, Chelsea Finn, Scott Niekum

专题命中偏好对齐：alignment(title,abstract)；RLHF(abstract)；分类 cs.CL、cs.AI、cs.LG

Comments 30 pages, 38th Conference on Neural Information Processing Systems (NeurIPS 2024)

2402.02416 2024-11-05 cs.CL cs.AI cs.LG 85%

Aligner: Efficient Alignment by Learning to Correct

Jiaming Ji, Boyuan Chen, Hantao Lou, Donghai Hong, Borong Zhang, Xuehai Pan, Juntao Dai, Tianyi Qiu, Yaodong Yang

专题命中偏好对齐：alignment(title,abstract)；harmlessness(abstract)；分类 cs.CL、cs.AI、cs.LG

Comments Accepted by NeurIPS 2024 Oral Presentation

2409.15360 2024-10-17 cs.LG cs.AI cs.CL 85%

Reward-Robust RLHF in LLMs

Yuzi Yan, Xingzhou Lou, Jialian Li, Yiping Zhang, Jian Xie, Chao Yu, Yu Wang, Dong Yan, Yuan Shen

专题命中偏好对齐：RLHF(title,abstract)；alignment(abstract)；分类 cs.CL、cs.AI、cs.LG

2406.11827 2024-10-07 cs.CL cs.AI cs.LG 85%

WPO: Enhancing RLHF with Weighted Preference Optimization

Wenxuan Zhou, Ravi Agrawal, Shujian Zhang, Sathish Reddy Indurthi, Sanqiang Zhao, Kaiqiang Song, Silei Xu, Chenguang Zhu

专题命中偏好对齐：RLHF(title,abstract)；DPO(abstract)；分类 cs.CL、cs.AI、cs.LG

Comments EMNLP 2024

2409.20370 2024-10-01 cs.LG cs.AI cs.CL 85%

The Perfect Blend: Redefining RLHF with Mixture of Judges

Tengyu Xu, Eryk Helenowski, Karthik Abinav Sankararaman, Di Jin, Kaiyan Peng, Eric Han, Shaoliang Nie, Chen Zhu, Hejia Zhang, Wenxuan Zhou, Zhouhao Zeng, Yun He, Karishma Mandyam, Arya Talabzadeh, Madian Khabsa, Gabriel Cohen, Yuandong Tian, Hao Ma, Sinong Wang, Han Fang

专题命中偏好对齐：RLHF(title,abstract)；DPO(abstract)；分类 cs.CL、cs.AI、cs.LG

Comments submitted to conference

2409.09603 2024-09-17 cs.AI cs.CL cs.LG 85%

Towards Data-Centric RLHF: Simple Metrics for Preference Dataset Comparison

Judy Hanwen Shen, Archit Sharma, Jun Qin

专题命中偏好对齐：RLHF(title,abstract)；alignment(abstract)；分类 cs.CL、cs.AI、cs.LG

Comments Working Paper

2403.07865 2024-09-17 cs.CL cs.AI cs.CR cs.LG cs.SE 85%

CodeAttack: Revealing Safety Generalization Challenges of Large Language Models via Code Completion

Qibing Ren, Chang Gao, Jing Shao, Junchi Yan, Xin Tan, Wai Lam, Lizhuang Ma

专题命中偏好对齐：safety(title,abstract)；alignment(abstract)；分类 cs.CL、cs.AI、cs.LG

Comments ACL Findings 2024, Code is available at https://github.com/renqibing/CodeAttack

2408.10270 2024-08-21 cs.LG cs.AI cs.CL 85%

SEAL: Systematic Error Analysis for Value ALignment

Manon Revel, Matteo Cargnelutti, Tyna Eloundou, Greg Leppert

专题命中偏好对齐：alignment(title,abstract)；RLHF(abstract)；分类 cs.CL、cs.AI、cs.LG

Comments 28 pages, 17 Figures, 8 Tables

2406.18682 2024-07-09 cs.CL cs.AI cs.LG 85%

The Multilingual Alignment Prism: Aligning Global and Local Preferences to Reduce Harm

Aakanksha, Arash Ahmadian, Beyza Ermis, Seraphina Goldfarb-Tarrant, Julia Kreutzer, Marzieh Fadaee, Sara Hooker

专题命中偏好对齐：alignment(title,abstract)；safety(abstract)；分类 cs.CL、cs.AI、cs.LG

2402.13228 2024-07-04 cs.CL cs.AI cs.LG 85%

Smaug: Fixing Failure Modes of Preference Optimisation with DPO-Positive

Arka Pal, Deep Karkhanis, Samuel Dooley, Manley Roberts, Siddartha Naidu, Colin White

专题命中偏好对齐：DPO(title,abstract)；alignment(abstract)；分类 cs.CL、cs.AI、cs.LG

2406.05534 2024-06-11 cs.AI cs.CL cs.LG 85%

Online DPO: Online Direct Preference Optimization with Fast-Slow Chasing

Biqing Qi, Pengfei Li, Fangyuan Li, Junqi Gao, Kaiyan Zhang, Bowen Zhou

专题命中偏好对齐：DPO(title,abstract)；alignment(abstract)；分类 cs.CL、cs.AI、cs.LG

2406.04274 2024-06-07 cs.LG cs.AI cs.CL 85%

Self-Play with Adversarial Critic: Provable and Scalable Offline Alignment for Language Models

Xiang Ji, Sanjeev Kulkarni, Mengdi Wang, Tengyang Xie

专题命中偏好对齐：alignment(title,abstract)；RLHF(abstract)；分类 cs.CL、cs.AI、cs.LG

2311.08045 2024-06-04 cs.CL cs.AI cs.LG 85%

Adversarial Preference Optimization: Enhancing Your Alignment via RM-LLM Game

Pengyu Cheng, Yifan Yang, Jian Li, Yong Dai, Tianhao Hu, Peixin Cao, Nan Du, Xiaolong Li

专题命中偏好对齐：alignment(title,abstract)；harmlessness(abstract)；分类 cs.CL、cs.AI、cs.LG

Comments Accepted by ACL2024 findings

2405.20053 2024-05-31 cs.CL cs.AI cs.LG 85%

Would I Lie To You? Inference Time Alignment of Language Models using Direct Preference Heads

Avelina Asada Hadji-Kyriacou, Ognjen Arandjelovic

专题命中偏好对齐：alignment(title)；RLHF(abstract)；DPO(abstract)；分类 cs.CL、cs.AI、cs.LG

2404.08495 2024-04-17 cs.LG cs.AI cs.CL 85%

Dataset Reset Policy Optimization for RLHF

Jonathan D. Chang, Wenhao Zhan, Owen Oertell, Kianté Brantley, Dipendra Misra, Jason D. Lee, Wen Sun

专题命中偏好对齐：RLHF(title,abstract)；DPO(abstract)；分类 cs.CL、cs.AI、cs.LG

Comments 28 pages, 6 tables, 3 Figures, 3 Algorithms

2312.15685 2024-04-17 cs.CL cs.AI cs.LG 85%

What Makes Good Data for Alignment? A Comprehensive Study of Automatic Data Selection in Instruction Tuning

Wei Liu, Weihao Zeng, Keqing He, Yong Jiang, Junxian He

专题命中偏好对齐：alignment(title,abstract)；DPO(abstract)；分类 cs.CL、cs.AI、cs.LG

Comments ICLR2024 Camera Ready. Data and model checkpoints are available at https://github.com/hkust-nlp/deita

2402.01694 2024-02-06 cs.CL cs.AI cs.LG 85%

ARGS: Alignment as Reward-Guided Search

Maxim Khanov, Jirayu Burapacheep, Yixuan Li

专题命中偏好对齐：alignment(title,abstract)；RLHF(abstract)；分类 cs.CL、cs.AI、cs.LG

Comments ICLR 2024

2304.06767 2023-12-04 cs.LG cs.AI cs.CL cs.CV stat.ML 85%

RAFT: Reward rAnked FineTuning for Generative Foundation Model Alignment

Hanze Dong, Wei Xiong, Deepanshu Goyal, Yihan Zhang, Winnie Chow, Rui Pan, Shizhe Diao, Jipeng Zhang, Kashun Shum, Tong Zhang

专题命中偏好对齐：alignment(title,abstract)；RLHF(abstract)；分类 cs.CL、cs.AI、cs.LG

Comments 29 pages, 12 figures, Published in Transactions on Machine Learning Research (TMLR)

2310.05344 2023-10-10 cs.CL cs.AI cs.LG 85%

SteerLM: Attribute Conditioned SFT as an (User-Steerable) Alternative to RLHF

Yi Dong, Zhilin Wang, Makesh Narsimhan Sreedhar, Xianchao Wu, Oleksii Kuchaiev

专题命中偏好对齐：RLHF(title,abstract)；alignment(abstract)；分类 cs.CL、cs.AI、cs.LG

Comments Findings of EMNLP 2023

2309.00754 2023-09-06 cs.LG cs.AI cs.CL 85%

Efficient RLHF: Reducing the Memory Usage of PPO

Michael Santacroce, Yadong Lu, Han Yu, Yuanzhi Li, Yelong Shen

专题命中偏好对齐：RLHF(title,abstract)；alignment(abstract)；分类 cs.CL、cs.AI、cs.LG

2307.04964 2023-07-19 cs.CL cs.AI cs.LG 85%

Secrets of RLHF in Large Language Models Part I: PPO

Rui Zheng, Shihan Dou, Songyang Gao, Yuan Hua, Wei Shen, Binghai Wang, Yan Liu, Senjie Jin, Qin Liu, Yuhao Zhou, Limao Xiong, Lu Chen, Zhiheng Xi, Nuo Xu, Wenbin Lai, Minghao Zhu, Cheng Chang, Zhangyue Yin, Rongxiang Weng, Wensen Cheng, Haoran Huang, Tianxiang Sun, Hang Yan, Tao Gui, Qi Zhang, Xipeng Qiu, Xuanjing Huang

专题命中偏好对齐：RLHF(title,abstract)；alignment(abstract)；分类 cs.CL、cs.AI、cs.LG

2603.03192 2026-03-31 cs.CV cs.CL cs.LG 84%

MoD-DPO: Towards Mitigating Cross-modal Hallucinations in Omni LLMs using Modality Decoupled Preference Optimization

MoD-DPO：通过模态解耦偏好优化缓解多模态幻觉

Ashutosh Chaubey, Jiacheng Pang, Mohammad Soleymani

机构 * University of Southern California（南加州大学）

专题命中偏好对齐：DPO(title,abstract)；alignment(abstract)；分类 cs.CL、cs.LG

AI总结本文提出MoD-DPO框架，通过引入模态感知正则化项和语言先验去偏惩罚，提升多模态大模型的模态对齐能力，实验表明其在多模态幻觉基准测试中表现优异。

Comments CVPR 2026. Project Page: https://mod-dpo.github.io/

2608.15530 2026-08-18 cs.CL 新提交 84%

Why Summaries Turn Neutral: Policy Attribution for Sentiment Drift in Reinforcement Learning from Human Feedback

为何摘要变得中立：人类反馈强化学习中情感漂移的策略归因

Mikhail Krasitskii, Alexander Gelbukh, Olga Kolesnikova, Grigori Sidorov

机构 * Instituto Politécnico Nacional (IPN)（墨西哥国立理工学院）； Centro de Investigación en Computación (CIC)（计算机研究中心）

专题命中偏好对齐：RLHF(summary_cn,abstract)；safety(abstract)；分类 cs.CL

AI总结本文针对RLHF引发摘要情感漂移的问题，提出Policy Attribution框架溯源漂移来源，验证了跨语言漂移特性，并提出感知情感的正则化技术以降低漂移，且相关代码将公开。

1. 偏好对齐 3252 篇

Regressing the Relative Future: Efficient Policy Optimization for Multi-turn RLHF

Improving LLM General Preference Alignment via Optimistic Online Mirror Descent

Reusing Embeddings: Reproducible Reward Model Research in Large Language Model Alignment without GPUs

Reviving The Classics: Active Reward Modeling in Large Language Model Alignment

3D-Properties: Identifying Challenges in DPO and Charting a Path Forward

Alignment faking in large language models

EACO: Enhancing Alignment in Multimodal LLMs via Critical Observation

Scaling Laws for Reward Model Overoptimization in Direct Alignment Algorithms

Aligner: Efficient Alignment by Learning to Correct

Reward-Robust RLHF in LLMs

WPO: Enhancing RLHF with Weighted Preference Optimization

The Perfect Blend: Redefining RLHF with Mixture of Judges

Towards Data-Centric RLHF: Simple Metrics for Preference Dataset Comparison

CodeAttack: Revealing Safety Generalization Challenges of Large Language Models via Code Completion

SEAL: Systematic Error Analysis for Value ALignment

The Multilingual Alignment Prism: Aligning Global and Local Preferences to Reduce Harm

Smaug: Fixing Failure Modes of Preference Optimisation with DPO-Positive

Online DPO: Online Direct Preference Optimization with Fast-Slow Chasing

Self-Play with Adversarial Critic: Provable and Scalable Offline Alignment for Language Models

Adversarial Preference Optimization: Enhancing Your Alignment via RM-LLM Game

Would I Lie To You? Inference Time Alignment of Language Models using Direct Preference Heads

Dataset Reset Policy Optimization for RLHF

What Makes Good Data for Alignment? A Comprehensive Study of Automatic Data Selection in Instruction Tuning

ARGS: Alignment as Reward-Guided Search

RAFT: Reward rAnked FineTuning for Generative Foundation Model Alignment

SteerLM: Attribute Conditioned SFT as an (User-Steerable) Alternative to RLHF

Efficient RLHF: Reducing the Memory Usage of PPO

Secrets of RLHF in Large Language Models Part I: PPO

MoD-DPO: Towards Mitigating Cross-modal Hallucinations in Omni LLMs using Modality Decoupled Preference Optimization

Why Summaries Turn Neutral: Policy Attribution for Sentiment Drift in Reinforcement Learning from Human Feedback