大模型对齐与安全 - arXivDaily 专题

2410.08067 2025-05-13 cs.LG cs.AI 84%

Reward-Augmented Data Enhances Direct Preference Alignment of LLMs

Shenao Zhang, Zhihan Liu, Boyi Liu, Yufeng Zhang, Yingxiang Yang, Yongfei Liu, Liyu Chen, Tao Sun, Zhaoran Wang

专题命中偏好对齐：alignment(title,abstract)；DPO(abstract)；分类 cs.AI、cs.LG

Comments Published at ICML 2025

2411.00062 2025-04-11 cs.CL cs.AI physics.data-an stat.ML 84%

Scalable Reinforcement Post-Training Beyond Static Human Prompts: Evolving Alignment via Asymmetric Self-Play

Ziyu Ye, Rishabh Agarwal, Tianqi Liu, Rishabh Joshi, Sarmishta Velury, Quoc V. Le, Qijun Tan, Yuan Liu

专题命中偏好对齐：alignment(title,abstract)；DPO(abstract)；分类 cs.CL、cs.AI

Comments spotlight @ neurips language gamification workshop. updated the problem description and added new online RL experiments in this version

2411.02442 2025-04-01 cs.CL cs.AI cs.IR 84%

TODO: Enhancing LLM Alignment with Ternary Preferences

Yuxiang Guo, Lu Yin, Bo Jiang, Jiaqi Zhang

专题命中偏好对齐：alignment(title,abstract)；DPO(abstract)；分类 cs.CL、cs.AI

Comments Accepted to ICLR 2025

2410.01532 2025-04-01 cs.CL cs.AI cs.CV cs.HC 84%

Seeing Eye to AI: Human Alignment via Gaze-Based Response Rewards for Large Language Models

Angela Lopez-Cardona, Carlos Segura, Alexandros Karatzoglou, Sergi Abadal, Ioannis Arapakis

专题命中偏好对齐：alignment(title,abstract)；RLHF(abstract)；分类 cs.CL、cs.AI

Comments This paper has been accepted to ICLR 2025

2410.12621 2025-03-26 cs.CL cs.LG 84%

Weak-to-Strong Generalization beyond Accuracy: a Pilot Study in Safety, Toxicity, and Legal Reasoning

Ruimeng Ye, Yang Xiao, Bo Hui

专题命中偏好对齐：safety(title,abstract)；alignment(abstract)；分类 cs.CL、cs.LG

2503.13538 2025-03-19 cs.LG cs.AI 84%

From Demonstrations to Rewards: Alignment Without Explicit Human Preferences

Siliang Zeng, Yao Liu, Huzefa Rangwala, George Karypis, Mingyi Hong, Rasool Fakoor

专题命中偏好对齐：alignment(title,abstract)；RLHF(abstract)；分类 cs.AI、cs.LG

2502.18548 2025-03-14 cs.LG cs.AI 84%

What is the Alignment Objective of GRPO?

Milan Vojnovic, Se-Young Yun

专题命中偏好对齐：alignment(title,abstract)；RLHF(abstract)；分类 cs.AI、cs.LG

2503.09925 2025-03-14 cs.LG cs.CL 84%

PluralLLM: Pluralistic Alignment in LLMs via Federated Learning

Mahmoud Srewa, Tianyu Zhao, Salma Elmalaki

专题命中偏好对齐：alignment(title,abstract)；RLHF(abstract)；分类 cs.CL、cs.LG

2503.03039 2025-03-06 cs.LG cs.AI 84%

LLM Misalignment via Adversarial RLHF Platforms

Erfan Entezami, Ali Naseh

专题命中偏好对齐：RLHF(title,abstract)；alignment(abstract)；分类 cs.AI、cs.LG

2410.07672 2025-03-04 cs.CL cs.AI 84%

MACPO: Weak-to-Strong Alignment via Multi-Agent Contrastive Preference Optimization

Yougang Lyu, Lingyong Yan, Zihan Wang, Dawei Yin, Pengjie Ren, Maarten de Rijke, Zhaochun Ren

专题命中偏好对齐：alignment(title,abstract)；RLHF(abstract)；分类 cs.CL、cs.AI

Comments ICLR 2025

2406.11431 2025-03-03 cs.CL cs.AI 84%

Super(ficial)-alignment: Strong Models May Deceive Weak Models in Weak-to-Strong Generalization

Wenkai Yang, Shiqi Shen, Guangyao Shen, Wei Yao, Yong Liu, Zhi Gong, Yankai Lin, Ji-Rong Wen

专题命中偏好对齐：alignment(title,abstract)；harmlessness(abstract)；分类 cs.CL、cs.AI

Comments Accepted at ICLR 2025, camera-ready version

2409.12914 2025-02-27 cs.LG cs.CL 84%

Evaluating Defences against Unsafe Feedback in RLHF

Domenic Rosati, Giles Edkins, Harsh Raj, David Atanasov, Subhabrata Majumdar, Janarthanan Rajendran, Frank Rudzicz, Hassan Sajjad

专题命中偏好对齐：RLHF(title,abstract)；safety(abstract)；分类 cs.CL、cs.LG

2404.09656 2025-02-26 cs.LG cs.CL 84%

Learn Your Reference Model for Real Good Alignment

Alexey Gorbatovski, Boris Shaposhnikov, Alexey Malakhov, Nikita Surnachev, Yaroslav Aksenov, Ian Maksimov, Nikita Balagansky, Daniil Gavrilov

专题命中偏好对齐：alignment(title,abstract)；DPO(abstract)；分类 cs.CL、cs.LG

2406.03642 2025-02-24 cs.CL cs.LG 84%

Is Free Self-Alignment Possible?

Dyah Adila, Changho Shin, Yijing Zhang, Frederic Sala

专题命中偏好对齐：alignment(title,abstract)；DPO(abstract)；分类 cs.CL、cs.LG

2502.00203 2025-02-11 cs.LG cs.CL 84%

Reward-aware Preference Optimization: A Unified Mathematical Framework for Model Alignment

Shengyang Sun, Yian Zhang, Alexander Bukharin, David Mosallanezhad, Jiaqi Zeng, Soumye Singhal, Gerald Shen, Adithya Renduchintala, Tugrul Konuk, Yi Dong, Zhilin Wang, Dmitry Chichkov, Olivier Delalleau, Oleksii Kuchaiev

专题命中偏好对齐：alignment(title,abstract)；DPO(abstract)；分类 cs.CL、cs.LG

Comments 8 pages, 4 figures; update author names

2502.04602 2025-02-10 cs.CL cs.AI 84%

Extracting and Understanding the Superficial Knowledge in Alignment

Runjin Chen, Gabriel Jacob Perin, Xuxi Chen, Xilun Chen, Yan Han, Nina S. T. Hirata, Junyuan Hong, Bhavya Kailkhura

专题命中偏好对齐：alignment(title,abstract)；safety(abstract)；分类 cs.CL、cs.AI

2501.19266 2025-02-03 cs.AI cs.LG econ.TH 84%

Jackpot! Alignment as a Maximal Lottery

Roberto-Rafael Maura-Rivero, Marc Lanctot, Francesco Visin, Kate Larson

专题命中偏好对齐：alignment(title,abstract)；RLHF(abstract)；分类 cs.AI、cs.LG

2404.01054 2025-01-30 cs.CL cs.AI 84%

Regularized Best-of-N Sampling with Minimum Bayes Risk Objective for Language Model Alignment

Yuu Jinnai, Tetsuro Morimura, Kaito Ariu, Kenshi Abe

专题命中偏好对齐：alignment(title)；RLHF(abstract)；DPO(abstract)；分类 cs.CL、cs.AI

Comments NAACL 2025

2409.15268 2025-01-28 cs.LG cs.AI 84%

Style Outweighs Substance: Failure Modes of LLM Judges in Alignment Benchmarking

Benjamin Feuer, Micah Goldblum, Teresa Datta, Sanjana Nambiar, Raz Besaleli, Samuel Dooley, Max Cembalest, John P. Dickerson

专题命中偏好对齐：alignment(title,abstract)；safety(abstract)；分类 cs.AI、cs.LG

Comments ICLR 2025

2405.15624 2025-01-28 cs.LG cs.AI 84%

Inverse-RLignment: Large Language Model Alignment from Demonstrations through Inverse Reinforcement Learning

Hao Sun, Mihaela van der Schaar

专题命中偏好对齐：alignment(title,abstract)；safety(abstract)；分类 cs.AI、cs.LG

2501.09254 2025-01-17 cs.LG cs.AI cs.GT 84%

Clone-Robust AI Alignment

Ariel D. Procaccia, Benjamin Schiffer, Shirley Zhang

专题命中偏好对齐：alignment(title,abstract)；RLHF(abstract)；分类 cs.AI、cs.LG

2501.03486 2025-01-08 cs.LG cs.AI 84%

Align-Pro: A Principled Approach to Prompt Optimization for LLM Alignment

Prashant Trivedi, Souradip Chakraborty, Avinash Reddy, Vaneet Aggarwal, Amrit Singh Bedi, George K. Atia

专题命中偏好对齐：alignment(title,abstract)；RLHF(abstract)；分类 cs.AI、cs.LG

Comments 27 pages, Accepted in AAAI 2025

2501.02869 2025-01-07 cs.CL cs.AI 84%

IIMedGPT: Promoting Large Language Model Capabilities of Medical Tasks by Efficient Human Preference Alignment

Yiming Zhang, Zheng Chang, Wentao Cai, MengXing Ren, Kang Yuan, Yining Sun, Zenghui Ding

专题命中偏好对齐：alignment(title,abstract)；DPO(abstract)；分类 cs.CL、cs.AI

2412.15453 2024-12-23 cs.CL cs.AI 84%

Northeastern Uni at Multilingual Counterspeech Generation: Enhancing Counter Speech Generation with LLM Alignment through Direct Preference Optimization

Sahil Wadhwa, Chengtian Xu, Haoming Chen, Aakash Mahalingam, Akankshya Kar, Divya Chaudhary

专题命中偏好对齐：alignment(title,abstract)；DPO(abstract)；分类 cs.CL、cs.AI

Comments 10 pages, 6 tables, 1 figure, The First Workshop on Multilingual Counterspeech Generation (MCG) at The 31st International Conference on Computational Linguistics (COLING 2025)

Journal ref The First Workshop on Multilingual Counterspeech Generation (MCG) at The 31st International Conference on Computational Linguistics (COLING 2025)

2412.15280 2024-12-23 cs.CL cs.AI cs.IR 84%

Context-DPO: Aligning Language Models for Context-Faithfulness

Baolong Bi, Shaohan Huang, Yiwei Wang, Tianchi Yang, Zihan Zhang, Haizhen Huang, Lingrui Mei, Junfeng Fang, Zehao Li, Furu Wei, Weiwei Deng, Feng Sun, Qi Zhang, Shenghua Liu

专题命中偏好对齐：DPO(title,abstract)；alignment(abstract)；分类 cs.CL、cs.AI

2412.14510 2024-12-20 cs.CL cs.AI 84%

PA-RAG: RAG Alignment via Multi-Perspective Preference Optimization

Jiayi Wu, Hengyi Cai, Lingyong Yan, Hao Sun, Xiang Li, Shuaiqiang Wang, Dawei Yin, Ming Gao

专题命中偏好对齐：alignment(title,abstract)；DPO(abstract)；分类 cs.CL、cs.AI

2412.13746 2024-12-19 cs.CL cs.AI cs.IR 84%

RAG-RewardBench: Benchmarking Reward Models in Retrieval Augmented Generation for Preference Alignment

Zhuoran Jin, Hongbang Yuan, Tianyi Men, Pengfei Cao, Yubo Chen, Kang Liu, Jun Zhao

专题命中偏好对齐：alignment(title,abstract)；trustworthy(abstract)；分类 cs.CL、cs.AI

Comments 26 pages, 12 figures, 6 tables

2406.08124 2024-12-19 cs.CL cs.AI 84%

Legend: Leveraging Representation Engineering to Annotate Safety Margin for Preference Datasets

Duanyu Feng, Bowen Qin, Chen Huang, Youcheng Huang, Zheng Zhang, Wenqiang Lei

专题命中偏好对齐：safety(title,abstract)；alignment(abstract)；分类 cs.CL、cs.AI

Comments Our code is available at https://github.com/colfeng/Legend

2412.09544 2024-12-13 cs.LG cs.AI math.OC math.ST stat.ML stat.TH 84%

Sail into the Headwind: Alignment via Robust Rewards and Dynamic Labels against Reward Hacking

Paria Rashidinejad, Yuandong Tian

专题命中偏好对齐：alignment(title,abstract)；DPO(abstract)；分类 cs.AI、cs.LG

Comments 46 pages, 3 figures

2405.16436 2024-12-05 cs.LG cs.AI stat.ML 84%

Provably Mitigating Overoptimization in RLHF: Your SFT Loss is Implicitly an Adversarial Regularizer

Zhihan Liu, Miao Lu, Shenao Zhang, Boyi Liu, Hongyi Guo, Yingxiang Yang, Jose Blanchet, Zhaoran Wang

专题命中偏好对齐：RLHF(title,abstract)；DPO(abstract)；分类 cs.AI、cs.LG

Comments Accepted by The Thirty-Eighth Annual Conference on Neural Information Processing Systems. 31 pages, 7 figures

1. 偏好对齐 3261 篇

Reward-Augmented Data Enhances Direct Preference Alignment of LLMs

Scalable Reinforcement Post-Training Beyond Static Human Prompts: Evolving Alignment via Asymmetric Self-Play

TODO: Enhancing LLM Alignment with Ternary Preferences

Seeing Eye to AI: Human Alignment via Gaze-Based Response Rewards for Large Language Models

Weak-to-Strong Generalization beyond Accuracy: a Pilot Study in Safety, Toxicity, and Legal Reasoning

From Demonstrations to Rewards: Alignment Without Explicit Human Preferences

What is the Alignment Objective of GRPO?

PluralLLM: Pluralistic Alignment in LLMs via Federated Learning

LLM Misalignment via Adversarial RLHF Platforms

MACPO: Weak-to-Strong Alignment via Multi-Agent Contrastive Preference Optimization

Super(ficial)-alignment: Strong Models May Deceive Weak Models in Weak-to-Strong Generalization

Evaluating Defences against Unsafe Feedback in RLHF

Learn Your Reference Model for Real Good Alignment

Is Free Self-Alignment Possible?

Reward-aware Preference Optimization: A Unified Mathematical Framework for Model Alignment

Extracting and Understanding the Superficial Knowledge in Alignment

Jackpot! Alignment as a Maximal Lottery

Regularized Best-of-N Sampling with Minimum Bayes Risk Objective for Language Model Alignment

Style Outweighs Substance: Failure Modes of LLM Judges in Alignment Benchmarking

Inverse-RLignment: Large Language Model Alignment from Demonstrations through Inverse Reinforcement Learning

Clone-Robust AI Alignment

Align-Pro: A Principled Approach to Prompt Optimization for LLM Alignment

IIMedGPT: Promoting Large Language Model Capabilities of Medical Tasks by Efficient Human Preference Alignment

Northeastern Uni at Multilingual Counterspeech Generation: Enhancing Counter Speech Generation with LLM Alignment through Direct Preference Optimization

Context-DPO: Aligning Language Models for Context-Faithfulness

PA-RAG: RAG Alignment via Multi-Perspective Preference Optimization

RAG-RewardBench: Benchmarking Reward Models in Retrieval Augmented Generation for Preference Alignment

Legend: Leveraging Representation Engineering to Annotate Safety Margin for Preference Datasets

Sail into the Headwind: Alignment via Robust Rewards and Dynamic Labels against Reward Hacking

Provably Mitigating Overoptimization in RLHF: Your SFT Loss is Implicitly an Adversarial Regularizer