大模型对齐与安全 - arXivDaily 专题

2409.09774 2024-11-07 cs.CV 82%

Generalizing Alignment Paradigm of Text-to-Image Generation with Preferences through $f$-divergence Minimization

Haoyuan Sun, Bo Xia, Yongzhe Chang, Xueqian Wang

专题命中偏好对齐：alignment(title,abstract)；DPO(abstract)

Comments 34 pages

2410.14872 2024-10-24 cs.LG cs.AI cs.CL 82%

How to Evaluate Reward Models for RLHF

Evan Frick, Tianle Li, Connor Chen, Wei-Lin Chiang, Anastasios N. Angelopoulos, Jiantao Jiao, Banghua Zhu, Joseph E. Gonzalez, Ion Stoica

专题命中偏好对齐：RLHF(title,abstract)；分类 cs.CL、cs.AI、cs.LG

2406.12168 2024-10-23 cs.LG cs.AI cs.CL 82%

BPO: Staying Close to the Behavior LLM Creates Better Online LLM Alignment

Wenda Xu, Jiachen Li, William Yang Wang, Lei Li

专题命中偏好对齐：alignment(title,abstract)；分类 cs.CL、cs.AI、cs.LG

Comments Wenda Xu and Jiachen Li contributed equally. Accepted by EMNLP 2024

2402.10207 2024-10-17 cs.LG cs.AI cs.CL 82%

Rewards-in-Context: Multi-objective Alignment of Foundation Models with Dynamic Preference Adjustment

Rui Yang, Xiaoman Pan, Feng Luo, Shuang Qiu, Han Zhong, Dong Yu, Jianshu Chen

专题命中偏好对齐：alignment(title,abstract)；分类 cs.CL、cs.AI、cs.LG

Comments Accepted by ICML 2024

2310.11523 2024-10-16 cs.LG cs.AI cs.CL 82%

Group Preference Optimization: Few-Shot Alignment of Large Language Models

Siyan Zhao, John Dang, Aditya Grover

专题命中偏好对齐：alignment(title,abstract)；分类 cs.CL、cs.AI、cs.LG

Comments accepted at ICLR 2024, code at https://github.com/jamqd/Group-Preference-Optimization

2312.11819 2024-10-15 cs.LG cs.AI cs.CL 82%

An Adaptive Placement and Parallelism Framework for Accelerating RLHF Training

Youshao Xiao, Zhenglei Zhou, Fagui Mao, Weichang Wu, Shangchun Zhao, Lin Ju, Lei Liang, Xiaolu Zhang, Jun Zhou

专题命中偏好对齐：RLHF(title,abstract)；分类 cs.CL、cs.AI、cs.LG

2410.07513 2024-10-11 cs.LG cs.AI cs.CL 82%

Evolutionary Contrastive Distillation for Language Model Alignment

Julian Katz-Samuels, Zheng Li, Hyokun Yun, Priyanka Nigam, Yi Xu, Vaclav Petricek, Bing Yin, Trishul Chilimbi

专题命中偏好对齐：alignment(title)；DPO(abstract)；分类 cs.CL、cs.AI、cs.LG

2402.03469 2024-10-08 cs.LG cs.AI cs.CL 82%

Rethinking the Role of Proxy Rewards in Language Model Alignment

Sungdong Kim, Minjoon Seo

专题命中偏好对齐：alignment(title,abstract)；分类 cs.CL、cs.AI、cs.LG

Comments Accepted to EMNLP 2024 main conference

2410.03212 2024-10-07 cs.IR 82%

Data-Efficient Massive Tool Retrieval: A Reinforcement Learning Approach for Query-Tool Alignment with Language Models

Yuxiang Zhang, Xin Fan, Junjie Wang, Chongxian Chen, Fan Mo, Tetsuya Sakai, Hayato Yamana

专题命中偏好对齐：alignment(title,abstract)；DPO(abstract)

2409.10164 2024-09-17 cs.LG cs.AI cs.CL 82%

Quantile Regression for Distributional Reward Models in RLHF

Nicolai Dorka

专题命中偏好对齐：RLHF(title,abstract)；分类 cs.CL、cs.AI、cs.LG

2408.06266 2024-09-17 cs.LG cs.AI cs.CL 82%

Anchored Preference Optimization and Contrastive Revisions: Addressing Underspecification in Alignment

Karel D'Oosterlinck, Winnie Xu, Chris Develder, Thomas Demeester, Amanpreet Singh, Christopher Potts, Douwe Kiela, Shikib Mehri

专题命中偏好对齐：alignment(title,abstract)；分类 cs.CL、cs.AI、cs.LG

2309.00267 2024-09-04 cs.CL cs.AI cs.LG 82%

RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedback

Harrison Lee, Samrat Phatale, Hassan Mansoor, Thomas Mesnard, Johan Ferret, Kellie Lu, Colton Bishop, Ethan Hall, Victor Carbune, Abhinav Rastogi, Sushant Prakash

专题命中偏好对齐：RLHF(title,abstract)；分类 cs.CL、cs.AI、cs.LG

Comments Presented at ICML 2024

Journal ref Proceedings of the 41st International Conference on Machine Learning, PMLR 235:26874-26901, 2024

2408.12579 2024-08-23 cs.CL cs.AI cs.HC cs.IR cs.LG 82%

RuleAlign: Making Large Language Models Better Physicians with Diagnostic Rule Alignment

Xiaohan Wang, Xiaoyan Yang, Yuqi Zhu, Yue Shen, Jian Wang, Peng Wei, Lei Liang, Jinjie Gu, Huajun Chen, Ningyu Zhang

专题命中偏好对齐：alignment(title,abstract)；分类 cs.CL、cs.AI、cs.LG

Comments Ongoing work

2404.10501 2024-08-22 cs.CV cs.AI cs.CL cs.LG 82%

Self-Supervised Visual Preference Alignment

Ke Zhu, Zheng Ge, Liang Zhao, Xiangyu Zhang

专题命中偏好对齐：alignment(title,abstract)；分类 cs.CL、cs.AI、cs.LG

Comments MM2024 oral

2404.05530 2024-08-07 cs.CL cs.AI cs.CR cs.LG 82%

Best-of-Venom: Attacking RLHF by Injecting Poisoned Preference Data

Tim Baumgärtner, Yang Gao, Dana Alon, Donald Metzler

专题命中偏好对齐：RLHF(title,abstract)；分类 cs.CL、cs.AI、cs.LG

2406.18676 2024-07-19 cs.CL cs.AI cs.LG 82%

Understand What LLM Needs: Dual Preference Alignment for Retrieval-Augmented Generation

Guanting Dong, Yutao Zhu, Chenghao Zhang, Zechen Wang, Zhicheng Dou, Ji-Rong Wen

专题命中偏好对齐：alignment(title,abstract)；分类 cs.CL、cs.AI、cs.LG

Comments Work in progress

2406.13173 2024-07-17 cs.CV cs.AI cs.CL cs.LG 82%

Biomedical Visual Instruction Tuning with Clinician Preference Alignment

Hejie Cui, Lingjun Mao, Xin Liang, Jieyu Zhang, Hui Ren, Quanzheng Li, Xiang Li, Carl Yang

专题命中偏好对齐：alignment(title,abstract)；分类 cs.CL、cs.AI、cs.LG

2403.13213 2024-07-08 cs.LG cs.CL cs.CY 82%

From Representational Harms to Quality-of-Service Harms: A Case Study on Llama 2 Safety Safeguards

Khaoula Chehbouni, Megha Roshan, Emmanuel Ma, Futian Andrew Wei, Afaf Taik, Jackie CK Cheung, Golnoosh Farnadi

专题命中偏好对齐：safety(title,abstract)；分类 cs.CL、cs.CY、cs.LG

Comments 9 pages, 4 figures. Accepted to Findings of the Association for Computational Linguistics: ACL 2024

2407.02552 2024-07-04 cs.CL cs.AI cs.LG 82%

RLHF Can Speak Many Languages: Unlocking Multilingual Preference Optimization for LLMs

John Dang, Arash Ahmadian, Kelly Marchisio, Julia Kreutzer, Ahmet Üstün, Sara Hooker

专题命中偏好对齐：RLHF(title)；alignment(abstract)；分类 cs.CL、cs.AI、cs.LG

2406.07971 2024-06-14 cs.CL cs.AI cs.LG 82%

It Takes Two: On the Seamlessness between Reward and Policy Model in RLHF

Taiming Lu, Lingfeng Shen, Xinyu Yang, Weiting Tan, Beidi Chen, Huaxiu Yao

专题命中偏好对齐：RLHF(title,abstract)；分类 cs.CL、cs.AI、cs.LG

2402.10958 2024-05-29 cs.CL cs.AI cs.LG 82%

Relative Preference Optimization: Enhancing LLM Alignment through Contrasting Responses across Identical and Diverse Prompts

Yueqin Yin, Zhendong Wang, Yi Gu, Hai Huang, Weizhu Chen, Mingyuan Zhou

专题命中偏好对齐：alignment(title)；DPO(abstract)；分类 cs.CL、cs.AI、cs.LG

2405.06639 2024-05-13 cs.LG cs.AI cs.CL 82%

Value Augmented Sampling for Language Model Alignment and Personalization

Seungwook Han, Idan Shenfeld, Akash Srivastava, Yoon Kim, Pulkit Agrawal

专题命中偏好对齐：alignment(title)；DPO(abstract)；分类 cs.CL、cs.AI、cs.LG

Comments Website: https://sites.google.com/view/llm-vas

2404.08555 2024-04-17 cs.LG cs.AI cs.CL 82%

RLHF Deciphered: A Critical Analysis of Reinforcement Learning from Human Feedback for LLMs

Shreyas Chaudhari, Pranjal Aggarwal, Vishvak Murahari, Tanmay Rajpurohit, Ashwin Kalyan, Karthik Narasimhan, Ameet Deshpande, Bruno Castro da Silva

专题命中偏好对齐：RLHF(title,abstract)；分类 cs.CL、cs.AI、cs.LG

2403.06326 2024-03-12 cs.CL cs.AI cs.LG 82%

From Instructions to Constraints: Language Model Alignment with Automatic Constraint Verification

Fei Wang, Chao Shang, Sarthak Jain, Shuai Wang, Qiang Ning, Bonan Min, Vittorio Castelli, Yassine Benajiba, Dan Roth

专题命中偏好对齐：alignment(title,abstract)；分类 cs.CL、cs.AI、cs.LG

2212.10529 2024-03-01 cs.CL cs.AI cs.CY 82%

Evaluating Psychological Safety of Large Language Models

Xingxuan Li, Yutong Li, Lin Qiu, Shafiq Joty, Lidong Bing

专题命中偏好对齐：safety(title,abstract)；分类 cs.CL、cs.AI、cs.CY

Comments Preprint. Under review

2310.06452 2024-02-20 cs.LG cs.AI cs.CL 82%

Understanding the Effects of RLHF on LLM Generalisation and Diversity

Robert Kirk, Ishita Mediratta, Christoforos Nalmpantis, Jelena Luketina, Eric Hambro, Edward Grefenstette, Roberta Raileanu

专题命中偏好对齐：RLHF(title,abstract)；分类 cs.CL、cs.AI、cs.LG

Comments Code available here: https://github.com/facebookresearch/rlfh-gen-div

2402.07319 2024-02-13 cs.LG cs.AI cs.CL 82%

ODIN: Disentangled Reward Mitigates Hacking in RLHF

Lichang Chen, Chen Zhu, Davit Soselia, Jiuhai Chen, Tianyi Zhou, Tom Goldstein, Heng Huang, Mohammad Shoeybi, Bryan Catanzaro

专题命中偏好对齐：RLHF(title,abstract)；分类 cs.CL、cs.AI、cs.LG

2401.16335 2024-01-30 cs.LG cs.AI cs.CL stat.ML 82%

Iterative Data Smoothing: Mitigating Reward Overfitting and Overoptimization in RLHF

Banghua Zhu, Michael I. Jordan, Jiantao Jiao

专题命中偏好对齐：RLHF(title,abstract)；分类 cs.CL、cs.AI、cs.LG

2308.01320 2023-08-04 cs.LG cs.AI cs.CL 82%

DeepSpeed-Chat: Easy, Fast and Affordable RLHF Training of ChatGPT-like Models at All Scales

Zhewei Yao, Reza Yazdani Aminabadi, Olatunji Ruwase, Samyam Rajbhandari, Xiaoxia Wu, Ammar Ahmad Awan, Jeff Rasley, Minjia Zhang, Conglong Li, Connor Holmes, Zhongzhu Zhou, Michael Wyatt, Molly Smith, Lev Kurilenko, Heyang Qin, Masahiro Tanaka, Shuai Che, Shuaiwen Leon Song, Yuxiong He

专题命中偏好对齐：RLHF(title,abstract)；分类 cs.CL、cs.AI、cs.LG

Comments 14 pages, 7 figures

2301.06859 2023-01-18 cs.HC cs.AI cs.CL cs.LG 82%

Methodological reflections for AI alignment research using human feedback

Thilo Hagendorff, Sarah Fabi

专题命中偏好对齐：alignment(title,abstract)；分类 cs.CL、cs.AI、cs.LG

1. 偏好对齐 3261 篇

Generalizing Alignment Paradigm of Text-to-Image Generation with Preferences through $f$-divergence Minimization

How to Evaluate Reward Models for RLHF

BPO: Staying Close to the Behavior LLM Creates Better Online LLM Alignment

Rewards-in-Context: Multi-objective Alignment of Foundation Models with Dynamic Preference Adjustment

Group Preference Optimization: Few-Shot Alignment of Large Language Models

An Adaptive Placement and Parallelism Framework for Accelerating RLHF Training

Evolutionary Contrastive Distillation for Language Model Alignment

Rethinking the Role of Proxy Rewards in Language Model Alignment

Data-Efficient Massive Tool Retrieval: A Reinforcement Learning Approach for Query-Tool Alignment with Language Models

Quantile Regression for Distributional Reward Models in RLHF

Anchored Preference Optimization and Contrastive Revisions: Addressing Underspecification in Alignment

RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedback

RuleAlign: Making Large Language Models Better Physicians with Diagnostic Rule Alignment

Self-Supervised Visual Preference Alignment

Best-of-Venom: Attacking RLHF by Injecting Poisoned Preference Data

Understand What LLM Needs: Dual Preference Alignment for Retrieval-Augmented Generation

Biomedical Visual Instruction Tuning with Clinician Preference Alignment

From Representational Harms to Quality-of-Service Harms: A Case Study on Llama 2 Safety Safeguards

RLHF Can Speak Many Languages: Unlocking Multilingual Preference Optimization for LLMs

It Takes Two: On the Seamlessness between Reward and Policy Model in RLHF

Relative Preference Optimization: Enhancing LLM Alignment through Contrasting Responses across Identical and Diverse Prompts

Value Augmented Sampling for Language Model Alignment and Personalization

RLHF Deciphered: A Critical Analysis of Reinforcement Learning from Human Feedback for LLMs

From Instructions to Constraints: Language Model Alignment with Automatic Constraint Verification

Evaluating Psychological Safety of Large Language Models

Understanding the Effects of RLHF on LLM Generalisation and Diversity

ODIN: Disentangled Reward Mitigates Hacking in RLHF

Iterative Data Smoothing: Mitigating Reward Overfitting and Overoptimization in RLHF

DeepSpeed-Chat: Easy, Fast and Affordable RLHF Training of ChatGPT-like Models at All Scales

Methodological reflections for AI alignment research using human feedback