大模型对齐与安全 - arXivDaily 专题

2506.03056 2025-06-04 cs.AI cs.CY cs.LG 75%

Corrigibility as a Singular Target: A Vision for Inherently Reliable Foundation Models

Ram Potham, Max Harms

机构 * Independent Researcher（独立研究者）； Machine Intelligence Research Institute（机器智能研究院）

专题命中偏好对齐：alignment(abstract)；safety(abstract)；分类 cs.AI、cs.CY、cs.LG

Comments Preprint. This work has been submitted to the Reliable and Responsible Foundation Models Workshop at ICML 2025 for review

2505.19964 2025-05-27 cs.LG cs.AI cs.CL cs.GT 75%

The Limits of Preference Data for Post-Training

Eric Zhao, Jessica Dai, Pranjal Awasthi

专题命中偏好对齐：RLHF(abstract)；safety(abstract)；分类 cs.CL、cs.AI、cs.LG

2505.01706 2025-05-06 cs.AI cs.CL cs.LG 75%

Inducing Robustness in a 2 Dimensional Direct Preference Optimization Paradigm

Sarvesh Shashidhar, Ritik, Nachiketa Patil, Suraj Racha, Ganesh Ramakrishnan

机构 * Centre for Machine Intelligence and Data Science, IIT Bombay（印度理工学院班加罗尔机器智能与数据科学中心）； Koita Centre for Digital Health, IIT Bombay（印度理工学院班加罗尔数字健康中心）； Department of Computer Science and Engineering, IIT Bombay（印度理工学院班加罗尔计算机科学与工程系）

专题命中偏好对齐：alignment(abstract)；DPO(abstract)；分类 cs.CL、cs.AI、cs.LG

Comments Updated abstract, algorithm and experimental results

2504.14439 2025-04-22 cs.LG cs.AI cs.CL 75%

LoRe: Personalizing LLMs via Low-Rank Reward Modeling

Avinandan Bose, Zhihan Xiong, Yuejie Chi, Simon Shaolei Du, Lin Xiao, Maryam Fazel

机构 * FAIR at Meta（Meta 的 FAIR 部门）； University of Washington（华盛顿大学）； Carnegie Mellon University（卡内基梅隆大学）

专题命中偏好对齐：alignment(abstract)；RLHF(abstract)；分类 cs.CL、cs.AI、cs.LG

2503.10965 2025-03-31 cs.AI cs.CL cs.LG 75%

Auditing language models for hidden objectives

Samuel Marks, Johannes Treutlein, Trenton Bricken, Jack Lindsey, Jonathan Marcus, Siddharth Mishra-Sharma, Daniel Ziegler, Emmanuel Ameisen, Joshua Batson, Tim Belonax, Samuel R. Bowman, Shan Carter, Brian Chen, Hoagy Cunningham, Carson Denison, Florian Dietz, Satvik Golechha, Akbir Khan, Jan Kirchner, Jan Leike, Austin Meek, Kei Nishimura-Gasparian, Euan Ong, Christopher Olah, Adam Pearce, Fabien Roger, Jeanne Salle, Andy Shih, Meg Tong, Drake Thomas, Kelley Rivoire, Adam Jermyn, Monte MacDiarmid, Tom Henighan, Evan Hubinger

专题命中偏好对齐：alignment(abstract)；RLHF(abstract)；分类 cs.CL、cs.AI、cs.LG

2412.05818 2025-03-26 cs.CV cs.AI cs.CL cs.LG cs.MM 75%

SILMM: Self-Improving Large Multimodal Models for Compositional Text-to-Image Generation

Leigang Qu, Haochuan Li, Wenjie Wang, Xiang Liu, Juncheng Li, Liqiang Nie, Tat-Seng Chua

专题命中偏好对齐：alignment(abstract)；DPO(abstract)；分类 cs.CL、cs.AI、cs.LG

Comments CVPR 2025 Camera-ready. Project page: https://silmm.github.io/

2503.10723 2025-03-17 cs.CL cs.AI cs.IR cs.LG 75%

RankPO: Preference Optimization for Job-Talent Matching

Yafei Zhang, Murray Wang, Yu Wang, Xiaohui Wang

专题命中偏好对齐：alignment(abstract)；DPO(abstract)；分类 cs.CL、cs.AI、cs.LG

Comments 15 pages, 3 figures, 7 tables

2501.00581 2025-02-25 cs.CL cs.AI cs.LG 75%

Are the Values of LLMs Structurally Aligned with Humans? A Causal Perspective

Yipeng Kang, Junqi Wang, Yexin Li, Mengmeng Wang, Wenming Tu, Quansen Wang, Hengli Li, Tingjun Wu, Xue Feng, Fangwei Zhong, Zilong Zheng

专题命中偏好对齐：alignment(abstract)；RLHF(abstract)；分类 cs.CL、cs.AI、cs.LG

2412.10400 2025-02-25 cs.CL cs.AI cs.LG 75%

Reinforcement Learning Enhanced LLMs: A Survey

Shuhe Wang, Shengyu Zhang, Jie Zhang, Runyi Hu, Xiaoya Li, Tianwei Zhang, Jiwei Li, Fei Wu, Guoyin Wang, Eduard Hovy

专题命中偏好对齐：RLHF(abstract)；DPO(abstract)；分类 cs.CL、cs.AI、cs.LG

2502.12206 2025-02-19 cs.AI cs.CL cs.LG 75%

Evaluating the Paperclip Maximizer: Are RL-Based Language Models More Likely to Pursue Instrumental Goals?

Yufei He, Yuexin Li, Jiaying Wu, Yuan Sui, Yulin Chen, Bryan Hooi

专题命中偏好对齐：alignment(abstract)；RLHF(abstract)；分类 cs.CL、cs.AI、cs.LG

2501.07886 2025-01-15 cs.LG cs.AI cs.CL 75%

Iterative Label Refinement Matters More than Preference Optimization under Weak Supervision

Yaowen Ye, Cassidy Laidlaw, Jacob Steinhardt

专题命中偏好对齐：RLHF(abstract)；DPO(abstract)；分类 cs.CL、cs.AI、cs.LG

Comments 22 pages, 10 figures

2404.00530 2025-01-09 cs.CL cs.AI cs.LG 75%

Comparing Bad Apples to Good Oranges: Aligning Large Language Models via Joint Preference Optimization

Hritik Bansal, Ashima Suvarna, Gantavya Bhatt, Nanyun Peng, Kai-Wei Chang, Aditya Grover

专题命中偏好对齐：alignment(abstract)；DPO(abstract)；分类 cs.CL、cs.AI、cs.LG

Comments 22 pages, 16 figures, 7 tables

2409.06691 2024-12-31 cs.LG cs.AI cs.CL 75%

Geometric-Averaged Preference Optimization for Soft Preference Labels

Hiroki Furuta, Kuang-Huei Lee, Shixiang Shane Gu, Yutaka Matsuo, Aleksandra Faust, Heiga Zen, Izzeddin Gur

专题命中偏好对齐：alignment(abstract)；DPO(abstract)；分类 cs.CL、cs.AI、cs.LG

Comments Accepted at NeurIPS 2024

2412.15244 2024-12-23 cs.CL cs.AI cs.LG 75%

MPPO: Multi Pair-wise Preference Optimization for LLMs with Arbitrary Negative Samples

Shuo Xie, Fangzhi Zhu, Jiahui Wang, Lulu Wen, Wei Dai, Xiaowei Chen, Junxiong Zhu, Kai Zhou, Bo Zheng

专题命中偏好对齐：RLHF(abstract)；DPO(abstract)；分类 cs.CL、cs.AI、cs.LG

Comments Accepted by COLING2025

2310.13008 2024-12-20 cs.LG cs.AI cs.CL 75%

DavIR: Data Selection via Implicit Reward for Large Language Models

Haotian Zhou, Tingkai Liu, Qianli Ma, Yufeng Zhang, Jianbo Yuan, Pengfei Liu, Yang You, Hongxia Yang

专题命中偏好对齐：alignment(abstract)；DPO(abstract)；分类 cs.CL、cs.AI、cs.LG

2406.10522 2024-12-19 cs.LG cs.AI cs.CL 75%

Humor in AI: Massive Scale Crowd-Sourced Preferences and Benchmarks for Cartoon Captioning

Jifan Zhang, Lalit Jain, Yang Guo, Jiayi Chen, Kuan Lok Zhou, Siddharth Suresh, Andrew Wagenmaker, Scott Sievert, Timothy Rogers, Kevin Jamieson, Robert Mankoff, Robert Nowak

专题命中偏好对齐：RLHF(abstract)；DPO(abstract)；分类 cs.CL、cs.AI、cs.LG

2412.02685 2024-12-04 cs.CL cs.AI cs.LG 75%

T-REG: Preference Optimization with Token-Level Reward Regularization

Wenxuan Zhou, Shujian Zhang, Lingxiao Zhao, Tao Meng

专题命中偏好对齐：alignment(abstract)；RLHF(abstract)；分类 cs.CL、cs.AI、cs.LG

2404.13846 2024-12-04 cs.LG cs.AI cs.CL 75%

Filtered Direct Preference Optimization

Tetsuro Morimura, Mitsuki Sakamoto, Yuu Jinnai, Kenshi Abe, Kaito Ariu

专题命中偏好对齐：RLHF(abstract)；DPO(abstract)；分类 cs.CL、cs.AI、cs.LG

Comments EMNLP 2024

2411.05875 2024-11-12 cs.LG cs.AI cs.CL 75%

Towards Improved Preference Optimization Pipeline: from Data Generation to Budget-Controlled Regularization

Zhuotong Chen, Fang Liu, Jennifer Zhu, Wanyu Du, Yanjun Qi

专题命中偏好对齐：alignment(abstract)；DPO(abstract)；分类 cs.CL、cs.AI、cs.LG

Comments 15 pages

2406.16235 2024-11-11 cs.CL cs.AI cs.CR cs.LG 75%

Preference Tuning For Toxicity Mitigation Generalizes Across Languages

Xiaochen Li, Zheng-Xin Yong, Stephen H. Bach

专题命中偏好对齐：DPO(abstract)；safety(abstract)；分类 cs.CL、cs.AI、cs.LG

Comments Findings of EMNLP 2024

2410.22891 2024-10-31 cs.LG cs.AI cs.CL 75%

VPO: Leveraging the Number of Votes in Preference Optimization

Jae Hyeon Cho, Minkyung Park, Byung-Jun Lee

专题命中偏好对齐：RLHF(abstract)；DPO(abstract)；分类 cs.CL、cs.AI、cs.LG

2401.12086 2024-10-28 cs.CL cs.AI cs.LG 75%

West-of-N: Synthetic Preferences for Self-Improving Reward Models

Alizée Pace, Jonathan Mallinson, Eric Malmi, Sebastian Krause, Aliaksei Severyn

专题命中偏好对齐：alignment(abstract)；RLHF(abstract)；分类 cs.CL、cs.AI、cs.LG

2401.16635 2024-10-23 cs.LG cs.AI cs.CL 75%

Improving Reinforcement Learning from Human Feedback with Efficient Reward Model Ensemble

Shun Zhang, Zhenfang Chen, Sunli Chen, Yikang Shen, Zhiqing Sun, Chuang Gan

专题命中偏好对齐：alignment(abstract)；RLHF(abstract)；分类 cs.CL、cs.AI、cs.LG

2405.19567 2024-10-11 cs.AI cs.CL cs.CV cs.LG 75%

Dr-LLaVA: Visual Instruction Tuning with Symbolic Clinical Grounding

Shenghuan Sun, Alexander Schubert, Gregory M. Goldgof, Zhiqing Sun, Thomas Hartvigsen, Atul J. Butte, Ahmed Alaa

专题命中偏好对齐：alignment(abstract)；RLHF(abstract)；分类 cs.CL、cs.AI、cs.LG

Comments Code available at: https://github.com/AlaaLab/Dr-LLaVA

2406.11839 2024-10-08 cs.CV cs.AI cs.CL cs.LG 75%

mDPO: Conditional Preference Optimization for Multimodal Large Language Models

Fei Wang, Wenxuan Zhou, James Y. Huang, Nan Xu, Sheng Zhang, Hoifung Poon, Muhao Chen

专题命中偏好对齐：alignment(abstract)；DPO(abstract)；分类 cs.CL、cs.AI、cs.LG

Comments Accepted to EMNLP 2024 Main Conference. Project website: https://feiwang96.github.io/mDPO

2403.10704 2024-09-16 cs.LG cs.AI cs.CL 75%

Parameter Efficient Reinforcement Learning from Human Feedback

Hakim Sidahmed, Samrat Phatale, Alex Hutcheson, Zhuonan Lin, Zhang Chen, Zac Yu, Jarvis Jin, Simral Chaudhary, Roman Komarytsia, Christiane Ahlheim, Yonghao Zhu, Bowen Li, Saravanan Ganesh, Bill Byrne, Jessica Hoffmann, Hassan Mansoor, Wei Li, Abhinav Rastogi, Lucas Dixon

专题命中偏好对齐：alignment(abstract)；RLHF(abstract)；分类 cs.CL、cs.AI、cs.LG

2409.02118 2024-09-05 cs.LG cs.AI cs.CL 75%

TSO: Self-Training with Scaled Preference Optimization

Kaihui Chen, Hao Yi, Qingyang Li, Tianyu Qi, Yulan Hu, Fuzheng Zhang, Yong Liu

专题命中偏好对齐：alignment(abstract)；DPO(abstract)；分类 cs.CL、cs.AI、cs.LG

2408.10075 2024-08-20 cs.LG cs.AI cs.CL cs.RO 75%

Personalizing Reinforcement Learning from Human Feedback with Variational Preference Learning

Sriyash Poddar, Yanming Wan, Hamish Ivison, Abhishek Gupta, Natasha Jaques

专题命中偏好对齐：alignment(abstract)；RLHF(abstract)；分类 cs.CL、cs.AI、cs.LG

Comments weirdlabuw.github.io/vpl

2305.18290 2024-07-31 cs.LG cs.AI cs.CL 75%

Direct Preference Optimization: Your Language Model is Secretly a Reward Model

Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D. Manning, Chelsea Finn

专题命中偏好对齐：RLHF(abstract)；DPO(abstract)；分类 cs.CL、cs.AI、cs.LG

2404.08699 2024-07-30 cs.CL cs.AI cs.LG 75%

PoliTune: Analyzing the Impact of Data Selection and Fine-Tuning on Economic and Political Biases in Large Language Models

Ahmed Agiza, Mohamed Mostagir, Sherief Reda

专题命中偏好对齐：alignment(abstract)；DPO(abstract)；分类 cs.CL、cs.AI、cs.LG

Comments AIES '24: Proceedings of the 2024 AAAI/ACM Conference on AI, Ethics, and Society

1. 偏好对齐 3266 篇

Corrigibility as a Singular Target: A Vision for Inherently Reliable Foundation Models

The Limits of Preference Data for Post-Training

Inducing Robustness in a 2 Dimensional Direct Preference Optimization Paradigm

LoRe: Personalizing LLMs via Low-Rank Reward Modeling

Auditing language models for hidden objectives

SILMM: Self-Improving Large Multimodal Models for Compositional Text-to-Image Generation

RankPO: Preference Optimization for Job-Talent Matching

Are the Values of LLMs Structurally Aligned with Humans? A Causal Perspective

Reinforcement Learning Enhanced LLMs: A Survey

Evaluating the Paperclip Maximizer: Are RL-Based Language Models More Likely to Pursue Instrumental Goals?

Iterative Label Refinement Matters More than Preference Optimization under Weak Supervision

Comparing Bad Apples to Good Oranges: Aligning Large Language Models via Joint Preference Optimization

Geometric-Averaged Preference Optimization for Soft Preference Labels

MPPO: Multi Pair-wise Preference Optimization for LLMs with Arbitrary Negative Samples

DavIR: Data Selection via Implicit Reward for Large Language Models

Humor in AI: Massive Scale Crowd-Sourced Preferences and Benchmarks for Cartoon Captioning

T-REG: Preference Optimization with Token-Level Reward Regularization

Filtered Direct Preference Optimization

Towards Improved Preference Optimization Pipeline: from Data Generation to Budget-Controlled Regularization

Preference Tuning For Toxicity Mitigation Generalizes Across Languages

VPO: Leveraging the Number of Votes in Preference Optimization

West-of-N: Synthetic Preferences for Self-Improving Reward Models

Improving Reinforcement Learning from Human Feedback with Efficient Reward Model Ensemble

Dr-LLaVA: Visual Instruction Tuning with Symbolic Clinical Grounding

mDPO: Conditional Preference Optimization for Multimodal Large Language Models

Parameter Efficient Reinforcement Learning from Human Feedback

TSO: Self-Training with Scaled Preference Optimization

Personalizing Reinforcement Learning from Human Feedback with Variational Preference Learning

Direct Preference Optimization: Your Language Model is Secretly a Reward Model

PoliTune: Analyzing the Impact of Data Selection and Fine-Tuning on Economic and Political Biases in Large Language Models