大模型对齐与安全

Noise Contrastive Alignment of Language Models with Explicit Rewards

Huayu Chen, Guande He, Lifan Yuan, Ganqu Cui, Hang Su, Jun Zhu

专题命中 偏好对齐 ：alignment(title,abstract)；DPO(abstract)；分类 cs.CL、cs.LG

Comments NeurIPS 2024

2D-DPO: Scaling Direct Preference Optimization with 2-Dimensional Supervision

Shilong Li, Yancheng He, Hui Huang, Xingyuan Bu, Jiaheng Liu, Hangyu Guo, Weixun Wang, Jihao Gu, Wenbo Su, Bo Zheng

专题命中 偏好对齐 ：DPO(title,abstract)；alignment(abstract)；分类 cs.CL、cs.AI

Comments The first four authors contributed equally, 25 pages

PopAlign: Diversifying Contrasting Patterns for a More Comprehensive Alignment

Zekun Moore Wang, Shawn Wang, Kang Zhu, Jiaheng Liu, Ke Xu, Jie Fu, Wangchunshu Zhou, Wenhao Huang

专题命中 偏好对齐 ：alignment(title,abstract)；RLHF(abstract)；分类 cs.CL、cs.AI

Comments 28 pages

Anchored Alignment for Self-Explanations Enhancement

Luis Felipe Villa-Arenas, Ata Nizamoglu, Qianli Wang, Sebastian Möller, Vera Schmitt

专题命中 偏好对齐 ：alignment(title,abstract)；DPO(abstract)；分类 cs.CL、cs.AI

The Accuracy Paradox in RLHF: When Better Reward Models Don't Yield Better Language Models

Yanjun Chen, Dawei Zhu, Yirong Sun, Xinghao Chen, Wei Zhang, Xiaoyu Shen

专题命中 偏好对齐 ：RLHF(title,abstract)；alignment(abstract)；分类 cs.CL、cs.AI

Comments 10 pages, 27 figures (including 18 in the appendix), submitted to EMNLP 2024

Progressively Label Enhancement for Large Language Model Alignment

Biao Liu, Ning Xu, Xin Geng

专题命中 偏好对齐 ：alignment(title,abstract)；RLHF(abstract)；分类 cs.CL、cs.AI

SYNFAC-EDIT: Synthetic Imitation Edit Feedback for Factual Alignment in Clinical Summarization

Prakamya Mishra, Zonghai Yao, Parth Vashisht, Feiyun Ouyang, Beining Wang, Vidhi Dhaval Mody, Hong Yu

专题命中 偏好对齐 ：alignment(title,abstract)；DPO(abstract)；分类 cs.CL、cs.AI

Comments Equal contribution for the first two authors; To appear in proceedings of the Main Conference on Empirical Methods in Natural Language Processing (EMNLP) 2024

Negating Negatives: Alignment with Human Negative Samples via Distributional Dispreference Optimization

Shitong Duan, Xiaoyuan Yi, Peng Zhang, Yan Liu, Zheng Liu, Tun Lu, Xing Xie, Ning Gu

专题命中 偏好对齐 ：alignment(title,abstract)；DPO(abstract)；分类 cs.CL、cs.AI

Comments Accepted by EMNLP 2024(Findings)

Elephant in the Room: Unveiling the Impact of Reward Model Quality in Alignment

Yan Liu, Xiaoyuan Yi, Xiaokang Chen, Jing Yao, Jingwei Yi, Daoguang Zan, Zheng Liu, Xing Xie, Tsung-Yi Ho

专题命中 偏好对齐 ：alignment(title,abstract)；RLHF(abstract)；分类 cs.CL、cs.AI

Preference Alignment Improves Language Model-Based TTS

Jinchuan Tian, Chunlei Zhang, Jiatong Shi, Hao Zhang, Jianwei Yu, Shinji Watanabe, Dong Yu

专题命中 偏好对齐 ：alignment(title,abstract)；DPO(abstract)；分类 cs.CL、cs.AI

What Makes and Breaks Safety Fine-tuning? A Mechanistic Study

Samyak Jain, Ekdeep Singh Lubana, Kemal Oksuz, Tom Joy, Philip H. S. Torr, Amartya Sanyal, Puneet K. Dokania

专题命中 偏好对齐 ：safety(title,abstract)；jailbreak(abstract)；分类 cs.CL、cs.LG

Comments Preprint

LIRE: listwise reward enhancement for preference alignment

Mingye Zhu, Yi Liu, Lei Zhang, Junbo Guo, Zhendong Mao

专题命中 偏好对齐 ：alignment(title,abstract)；RLHF(abstract)；分类 cs.CL、cs.LG

Comments Accepted by ACL 2024 Findings

Direct Alignment of Language Models via Quality-Aware Self-Refinement

Runsheng Yu, Yong Wang, Xiaoqi Jiao, Youzhi Zhang, James T. Kwok

专题命中 偏好对齐 ：alignment(title)；RLHF(abstract)；DPO(abstract)；分类 cs.CL、cs.AI

Self-Augmented Preference Optimization: Off-Policy Paradigms for Language Model Alignment

Yueqin Yin, Zhendong Wang, Yujia Xie, Weizhu Chen, Mingyuan Zhou

专题命中 偏好对齐 ：alignment(title,abstract)；DPO(abstract)；分类 cs.CL、cs.LG

Understanding the performance gap between online and offline alignment algorithms

Yunhao Tang, Daniel Zhaohan Guo, Zeyu Zheng, Daniele Calandriello, Yuan Cao, Eugene Tarassov, Rémi Munos, Bernardo Ávila Pires, Michal Valko, Yong Cheng, Will Dabney

专题命中 偏好对齐 ：alignment(title,abstract)；RLHF(abstract)；分类 cs.AI、cs.LG

MetaRM: Shifted Distributions Alignment via Meta-Learning

Shihan Dou, Yan Liu, Enyu Zhou, Tianlong Li, Haoxiang Jia, Limao Xiong, Xin Zhao, Junjie Ye, Rui Zheng, Tao Gui, Qi Zhang, Xuanjing Huang

专题命中 偏好对齐 ：alignment(title,abstract)；RLHF(abstract)；分类 cs.CL、cs.LG

Comments 11 pages, 6 figures. arXiv admin note: text overlap with arXiv:2401.06080

Leveraging Domain Knowledge for Efficient Reward Modelling in RLHF: A Case-Study in E-Commerce Opinion Summarization

Swaroop Nath, Tejpalsingh Siledar, Sankara Sri Raghava Ravindra Muddu, Rupasai Rangaraju, Harshad Khadilkar, Pushpak Bhattacharyya, Suman Banerjee, Amey Patil, Sudhanshu Shekhar Singh, Muthusamy Chelliah, Nikesh Garera

专题命中 偏好对齐 ：RLHF(title,abstract)；alignment(abstract)；分类 cs.CL、cs.LG

Comments 19 pages, 6 figures, 21 tables

Distributional Preference Learning: Understanding and Accounting for Hidden Context in RLHF

Anand Siththaranjan, Cassidy Laidlaw, Dylan Hadfield-Menell

专题命中 偏好对齐 ：RLHF(title,abstract)；jailbreak(abstract)；分类 cs.AI、cs.LG

Comments Presented at ICLR 2024

Configurable Safety Tuning of Language Models with Synthetic Preference Data

Victor Gallego

专题命中 偏好对齐 ：safety(title,abstract)；DPO(abstract)；分类 cs.CL、cs.AI

RLCD: Reinforcement Learning from Contrastive Distillation for Language Model Alignment

Kevin Yang, Dan Klein, Asli Celikyilmaz, Nanyun Peng, Yuandong Tian

专题命中 偏好对齐 ：alignment(title,abstract)；harmlessness(abstract)；分类 cs.CL、cs.AI

Comments ICLR 2024

Preference Ranking Optimization for Human Alignment

Feifan Song, Bowen Yu, Minghao Li, Haiyang Yu, Fei Huang, Yongbin Li, Houfeng Wang

专题命中 偏好对齐 ：alignment(title,abstract)；RLHF(abstract)；分类 cs.CL、cs.AI

Comments Accepted by AAAI 2024

ICDPO: Effectively Borrowing Alignment Capability of Others via In-context Direct Preference Optimization

Feifan Song, Yuxuan Fan, Xin Zhang, Peiyi Wang, Houfeng Wang

专题命中 偏好对齐 ：alignment(title,abstract)；DPO(abstract)；分类 cs.CL、cs.AI

Aligning Large Language Models with Counterfactual DPO

Bradley Butcher

专题命中 偏好对齐 ：DPO(title,abstract)；alignment(abstract)；分类 cs.CL、cs.AI

The Unlocking Spell on Base LLMs: Rethinking Alignment via In-Context Learning

Bill Yuchen Lin, Abhilasha Ravichander, Ximing Lu, Nouha Dziri, Melanie Sclar, Khyathi Chandu, Chandra Bhagavatula, Yejin Choi

专题命中 偏好对齐 ：alignment(title,abstract)；RLHF(abstract)；分类 cs.CL、cs.AI

Comments 26 pages, 8 figures. Project website: https://allenai.github.io/re-align/

Data-Efficient Alignment of Large Language Models with Human Feedback Through Natural Language

Di Jin, Shikib Mehri, Devamanyu Hazarika, Aishwarya Padmakumar, Sungjin Lee, Yang Liu, Mahdi Namazifar

专题命中 偏好对齐 ：alignment(title,abstract)；RLHF(abstract)；分类 cs.CL、cs.AI

Comments Accepted by Workshop on Instruction Tuning and Instruction Following at NeurIPS 2023, Submitted to AAAI 2024

OpenAssistant Conversations -- Democratizing Large Language Model Alignment

Andreas Köpf, Yannic Kilcher, Dimitri von Rütte, Sotiris Anagnostidis, Zhi-Rui Tam, Keith Stevens, Abdullah Barhoum, Nguyen Minh Duc, Oliver Stanley, Richárd Nagyfi, Shahul ES, Sameer Suri, David Glushkov, Arnav Dantuluri, Andrew Maguire, Christoph Schuhmann, Huu Nguyen, Alexander Mattick

专题命中 偏好对齐 ：alignment(title,abstract)；RLHF(abstract)；分类 cs.CL、cs.AI

Comments Published in NeurIPS 2023 Datasets and Benchmarks

Zephyr: Direct Distillation of LM Alignment

Lewis Tunstall, Edward Beeching, Nathan Lambert, Nazneen Rajani, Kashif Rasul, Younes Belkada, Shengyi Huang, Leandro von Werra, Clémentine Fourrier, Nathan Habib, Nathan Sarrazin, Omar Sanseviero, Alexander M. Rush, Thomas Wolf

专题命中 偏好对齐 ：alignment(title,abstract)；RLHF(abstract)；分类 cs.CL、cs.LG

CycleAlign: Iterative Distillation from Black-box LLM to White-box Models for Better Human Alignment

Jixiang Hong, Quan Tu, Changyu Chen, Xing Gao, Ji Zhang, Rui Yan

专题命中 偏好对齐 ：alignment(title,abstract)；RLHF(abstract)；分类 cs.CL、cs.AI

Rewarded soups: towards Pareto-optimal alignment by interpolating weights fine-tuned on diverse rewards

Alexandre Ramé, Guillaume Couairon, Mustafa Shukor, Corentin Dancette, Jean-Baptiste Gaya, Laure Soulier, Matthieu Cord

专题命中 偏好对齐 ：alignment(title,abstract)；RLHF(abstract)；分类 cs.AI、cs.LG

Representative Social Choice: From Learning Theory to AI Alignment

Tianyi Qiu

机构 * Peking University（北京大学） ； UC Berkeley（加州大学伯克利分校） ； Center for Human-Compatible AI（人类兼容人工智能中心） 纠错

专题命中 偏好对齐 ：alignment(title,abstract)；分类 cs.CL、cs.AI、cs.CY

Comments Journal of Artificial Intelligence Research, in press. Best Paper at NeurIPS 2024 Pluralistic Alignment Workshop