大模型对齐与安全 - arXivDaily 专题

2505.08849 2025-05-15 cs.CR cs.AI cs.LG 86%

Improved Algorithms for Differentially Private Language Model Alignment

Keyu Chen, Hao Tang, Qinglin Liu, Yizhao Xu

机构 * Peking University（北京大学）

专题命中偏好对齐：alignment(title,abstract)；RLHF(abstract)；DPO(abstract)；分类 cs.AI、cs.LG

2503.22137 2025-03-31 cs.AI cs.LG 86%

Sharpe Ratio-Guided Active Learning for Preference Optimization in RLHF

Syrine Belakaria, Joshua Kazdan, Charles Marx, Chris Cundy, Willie Neiswanger, Sanmi Koyejo, Barbara E. Engelhardt, Stefano Ermon

专题命中偏好对齐：RLHF(title,abstract)；alignment(abstract)；DPO(abstract)；分类 cs.AI、cs.LG

2503.21720 2025-03-28 cs.CL cs.AI 86%

Collab: Controlled Decoding using Mixture of Agents for LLM Alignment

Souradip Chakraborty, Sujay Bhatt, Udari Madhushani Sehwag, Soumya Suvra Ghosal, Jiahao Qiu, Mengdi Wang, Dinesh Manocha, Furong Huang, Alec Koppel, Sumitra Ganesh

专题命中偏好对齐：alignment(title,abstract)；RLHF(abstract)；trustworthy(abstract)；分类 cs.CL、cs.AI

Comments Accepted to ICLR 2025

2312.00267 2025-03-21 cs.LG cs.AI stat.ML 86%

Sample Efficient Preference Alignment in LLMs via Active Exploration

Viraj Mehta, Syrine Belakaria, Vikramjeet Das, Ojash Neopane, Yijia Dai, Ilija Bogunovic, Barbara Engelhardt, Stefano Ermon, Jeff Schneider, Willie Neiswanger

专题命中偏好对齐：alignment(title,abstract)；RLHF(abstract)；DPO(abstract)；分类 cs.AI、cs.LG

2406.09760 2025-03-10 cs.CL cs.LG 86%

Bootstrapping Language Models with DPO Implicit Rewards

Changyu Chen, Zichen Liu, Chao Du, Tianyu Pang, Qian Liu, Arunesh Sinha, Pradeep Varakantham, Min Lin

专题命中偏好对齐：DPO(title,abstract)；alignment(abstract)；RLHF(abstract)；分类 cs.CL、cs.LG

Comments Accepted in ICLR 2025

2412.16974 2024-12-24 cs.AI cs.CL 86%

Cannot or Should Not? Automatic Analysis of Refusal Composition in IFT/RLHF Datasets and Refusal Behavior of Black-Box LLMs

Alexander von Recum, Christoph Schnabl, Gabor Hollbeck, Silas Alberti, Philip Blinde, Marvin von Hagen

专题命中偏好对齐：RLHF(title,abstract)；safety(abstract)；AI safety(abstract)；分类 cs.CL、cs.AI

Comments NeurIPS 2024 Workshop SFLLM

2412.13862 2024-12-19 cs.LG cs.CL 86%

Energy-Based Preference Model Offers Better Offline Alignment than the Bradley-Terry Preference Model

Yuzhong Hong, Hanshan Zhang, Junwei Bao, Hongfei Jiang, Yang Song

专题命中偏好对齐：alignment(title,abstract)；RLHF(abstract)；DPO(abstract)；分类 cs.CL、cs.LG

2405.19332 2024-11-06 cs.LG cs.AI 86%

Self-Exploring Language Models: Active Preference Elicitation for Online Alignment

Shenao Zhang, Donghan Yu, Hiteshi Sharma, Han Zhong, Zhihan Liu, Ziyi Yang, Shuohang Wang, Hany Hassan, Zhaoran Wang

专题命中偏好对齐：alignment(title,abstract)；RLHF(abstract)；DPO(abstract)；分类 cs.AI、cs.LG

2406.00832 2024-11-05 cs.CL cs.LG 86%

BoNBoN Alignment for Large Language Models and the Sweetness of Best-of-n Sampling

Lin Gui, Cristina Gârbacea, Victor Veitch

专题命中偏好对齐：alignment(title,abstract)；RLHF(abstract)；DPO(abstract)；分类 cs.CL、cs.LG

2406.11715 2024-10-28 cs.LG cs.CL cs.SE 86%

Measuring memorization in RLHF for code completion

Aneesh Pappu, Billy Porter, Ilia Shumailov, Jamie Hayes

专题命中偏好对齐：RLHF(title,abstract)；alignment(abstract)；DPO(abstract)；分类 cs.CL、cs.LG

2410.09362 2024-10-15 cs.LG cs.AI 86%

SeRA: Self-Reviewing and Alignment of Large Language Models using Implicit Reward Margins

Jongwoo Ko, Saket Dingliwal, Bhavana Ganesh, Sailik Sengupta, Sravan Bodapati, Aram Galstyan

专题命中偏好对齐：alignment(title,abstract)；RLHF(abstract)；DPO(abstract)；分类 cs.AI、cs.LG

2409.00858 2024-09-06 cs.RO cs.AI cs.HC cs.LG 86%

Trustworthy Human-AI Collaboration: Reinforcement Learning with Human Feedback and Physics Knowledge for Safe Autonomous Driving

Zilin Huang, Zihao Sheng, Sikai Chen

专题命中偏好对齐：trustworthy(title,abstract)；RLHF(abstract)；safety(abstract)；分类 cs.AI、cs.LG

Comments 33 pages, 20 figures

2304.11082 2024-06-04 cs.CL cs.AI 86%

Fundamental Limitations of Alignment in Large Language Models

Yotam Wolf, Noam Wies, Oshri Avnery, Yoav Levine, Amnon Shashua

专题命中偏好对齐：alignment(title,abstract)；safety(abstract)；AI safety(abstract)；分类 cs.CL、cs.AI

2405.17931 2024-05-29 cs.CL cs.LG 86%

Online Merging Optimizers for Boosting Rewards and Mitigating Tax in Alignment

Keming Lu, Bowen Yu, Fei Huang, Yang Fan, Runji Lin, Chang Zhou

专题命中偏好对齐：alignment(title,abstract)；RLHF(abstract)；DPO(abstract)；分类 cs.CL、cs.LG

2404.00486 2024-04-02 cs.CL cs.AI 86%

Dialectical Alignment: Resolving the Tension of 3H and Security Threats of LLMs

Shu Yang, Jiayuan Su, Han Jiang, Mengdi Li, Keyuan Cheng, Muhammad Asif Ali, Lijie Hu, Di Wang

专题命中偏好对齐：alignment(title,abstract)；RLHF(abstract)；DPO(abstract)；分类 cs.CL、cs.AI

2403.08309 2024-03-15 cs.LG cs.AI 86%

HRLAIF: Improvements in Helpfulness and Harmlessness in Open-domain Reinforcement Learning From AI Feedback

Ang Li, Qiugen Xiao, Peng Cao, Jian Tang, Yi Yuan, Zijie Zhao, Xiaoyuan Chen, Liang Zhang, Xiangyang Li, Kaitong Yang, Weidong Guo, Yukang Gan, Xu Yu, Daniell Wang, Ying Shan

专题命中偏好对齐：harmlessness(title,abstract)；RLHF(abstract)；red teaming(abstract)；分类 cs.AI、cs.LG

Comments 18 pages, 7 figures

2403.08635 2024-03-14 cs.LG cs.AI stat.ML 86%

Human Alignment of Large Language Models through Online Preference Optimisation

Daniele Calandriello, Daniel Guo, Remi Munos, Mark Rowland, Yunhao Tang, Bernardo Avila Pires, Pierre Harvey Richemond, Charline Le Lan, Michal Valko, Tianqi Liu, Rishabh Joshi, Zeyu Zheng, Bilal Piot

专题命中偏好对齐：alignment(title,abstract)；RLHF(abstract)；DPO(abstract)；分类 cs.AI、cs.LG

2402.04792 2024-03-04 cs.AI cs.CL cs.HC 86%

Direct Language Model Alignment from Online AI Feedback

Shangmin Guo, Biao Zhang, Tianlin Liu, Tianqi Liu, Misha Khalman, Felipe Llinares, Alexandre Rame, Thomas Mesnard, Yao Zhao, Bilal Piot, Johan Ferret, Mathieu Blondel

专题命中偏好对齐：alignment(title,abstract)；RLHF(abstract)；DPO(abstract)；分类 cs.CL、cs.AI

Comments 18 pages, 9 figures, 4 tables

2401.13136 2024-01-25 cs.CL cs.AI 86%

The Language Barrier: Dissecting Safety Challenges of LLMs in Multilingual Contexts

Lingfeng Shen, Weiting Tan, Sihao Chen, Yunmo Chen, Jingyu Zhang, Haoran Xu, Boyuan Zheng, Philipp Koehn, Daniel Khashabi

专题命中偏好对齐：safety(title,abstract)；alignment(abstract)；RLHF(abstract)；分类 cs.CL、cs.AI

2311.12908 2023-11-23 cs.CV cs.AI cs.GR cs.LG 86%

Diffusion Model Alignment Using Direct Preference Optimization

Bram Wallace, Meihua Dang, Rafael Rafailov, Linqi Zhou, Aaron Lou, Senthil Purushwalkam, Stefano Ermon, Caiming Xiong, Shafiq Joty, Nikhil Naik

专题命中偏好对齐：alignment(title,abstract)；RLHF(abstract)；DPO(abstract)；分类 cs.AI、cs.LG

2410.04350 2025-04-16 cs.CL 86%

TIS-DPO: Token-level Importance Sampling for Direct Preference Optimization With Estimated Weights

Aiwei Liu, Haoping Bai, Zhiyun Lu, Yanchao Sun, Xiang Kong, Simon Wang, Jiulong Shan, Albin Madappally Jose, Xiaojiang Liu, Lijie Wen, Philip S. Yu, Meng Cao

专题命中偏好对齐：DPO(title,abstract)；alignment(abstract)；harmlessness(abstract)；分类 cs.CL

Comments Published in ICLR 2025, code in https://github.com/exlaw/TIS-DPO

2606.24937 2026-07-28 cs.AI cs.CL cs.IR cs.LG 版本更新 86%

The Hitchhiker's Guide to Agentic AI: From Foundations to Systems

《智能体AI的搭车指南：从基础到系统》

Haggai Roitman

机构 * Haggai Roitman

专题命中偏好对齐：RLHF(abstract,abstract_cn)；DPO(abstract,abstract_cn)；alignment(abstract)；分类 cs.CL、cs.AI、cs.LG

AI总结本文为构建自主AI系统提供全面实践参考，覆盖从Transformer架构、训练优化到对齐、推理、检索增强生成、记忆系统、智能体设计模式及多智能体协调的完整技术栈。

Comments version 1.3

2606.11712 2026-06-11 cs.CL cs.AI cs.LG 新提交 86%

Substrate Asymmetry in User-Side Memory: A Diagnostic Framework

用户侧记忆中的子模块不对称性：一个诊断框架

Youwang Deng

机构 * EpistemicaLab — Independent Research（EpistemicaLab — 独立研究）

专题命中偏好对齐：RLHF(summary_cn,abstract)；alignment(abstract)；分类 cs.CL、cs.AI、cs.LG

AI总结提出一个诊断框架，将LLM用户侧记忆分解为行为一致性、事实存在和事实缺失三个正交子模块，发现参数记忆与检索记忆在不同子模块上存在不对称性，且RLHF调优加剧了这种不对称性。

Comments Preprint. Code: https://github.com/EpistemicaLab/substrate-asymmetry-memory

2507.21183 2026-05-11 cs.LG cs.AI cs.CL 86%

MaPPO: Maximum a Posteriori Preference Optimization with Prior Knowledge

MaPPO：结合先验知识的最大后验偏好优化

Guangchen Lan, Sipeng Zhang, Tianle Wang, Yuwei Zhang, Daoan Zhang, Xinpeng Wei, Xiaoman Pan, Hongming Zhang, Dong-Jun Han, Christopher G. Brinton

机构 * Purdue University（普渡大学）； University of California, San Diego（加州大学圣地亚哥分校）； Amazon（亚马逊）； Meta, FAIR ； Yonsei University（延世大学）

专题命中偏好对齐：DPO(summary_cn,abstract)；alignment(abstract)；分类 cs.CL、cs.AI、cs.LG

AI总结 MaPPO通过整合先验奖励知识优化偏好学习，提升对齐性能，无需额外超参数，适用于离线和在线设置，支持DPO变体插件，实验显示在多个基准上效果显著。

2410.15334 2025-06-09 cs.CV 86%

Modality-Fair Preference Optimization for Trustworthy MLLM Alignment

Songtao Jiang, Yan Zhang, Ruizhe Chen, Tianxiang Hu, Yeying Jin, Qinglin He, Yang Feng, Jian Wu, Zuozhu Liu

机构 * Zhejiang University（浙江大学）； ByteDance（字节跳动）； National University of Singapore（新加坡国立大学）； Angelalign Inc., China（中国Angelalign公司）

专题命中偏好对齐：alignment(title,abstract)；trustworthy(title)

2608.13069 2026-08-14 cs.AI 新提交 85%

Behavioral Reprogramming of Open-Weights Models: Cognitive Plasticity and Alignment Bounds

开放权重模型的行为重编程：认知可塑性与对齐边界

Lucia Malíčková

机构 * National Supercomputing Centre, Slovakia（斯洛伐克国家超级计算中心）

专题命中偏好对齐：alignment(title,abstract)；DPO(abstract,abstract_cn)；分类 cs.AI

AI总结该研究通过大规模并行超参数搜索等方法，对开放权重LLMs进行行为重编程，实现了主动苏格拉底式对话框架，明确了PEFT的边界等关键结论，为跨语言行为修改提供了实证框架。

Comments Preprint submitted to arXiv, August 12, 2026. 13 pages, 5 figures

2604.25077 2026-08-05 cs.AI 版本更新 85%

Evaluating Risks in Weak-to-Strong Alignment: A Bias-Variance Perspective

在弱到强对齐中评估风险：从偏差-方差视角出发

Hamid Osooli, Kareema Batool, Rick Gentry, Tiasa Singha Roy, Ashwin Gupta, Anirudha Ramesh

机构 * University of Illinois Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）； Microsoft（微软）； InstaDeep ； New York University（纽约大学）

专题命中偏好对齐：alignment(title,abstract)；RLHF(abstract,abstract_cn)；分类 cs.AI

AI总结本文从偏差-方差视角分析弱到强对齐的风险，通过连续置信度分数研究经验组件，发现强模型方差是欺骗的主要预测因素，表明弱强依赖性的重要性。

2607.13430 2026-07-16 cs.CL 新提交 85%

Exploring Post-Training Alignment of Small Language Models for Biomedical Data-to-Text Generation: A Case Study of Medication Leaflet

探索用于生物医学数据到文本生成的小语言模型的训练后对齐：以药品说明书为例

Xi Yang, Guodong Liu, Chuqin Li, Fan Wu, Ergin Soysal, Min Jiang, Xing He, Jiang Bian, Yi Guo, Shams Zaman, Thomas Fuchs, Todd Sanger, Yonghui Wu

专题命中偏好对齐：alignment(title,abstract)；DPO(abstract,abstract_cn)；分类 cs.CL

AI总结该研究对生物医学数据到文本生成任务训练小语言模型的方法进行比较分析，在药品说明书数据集上用基于Qwen的SLMs探索多种训练后方法，通过整理数据评估模型，结果显示对齐后的SLMs性能优异，GRPO跨数据集性能最稳健。

Comments 10 pages, 1 figures

2512.15765 2026-07-07 cs.LG cs.GT stat.ML 版本更新 85%

Shapley-based Data Valuation for LLM Alignment via Sequential Preference Optimization

基于Shapley值的大语言模型对齐数据估值：通过顺序偏好优化

Mélissa Tamine, Otmane Sakhi, Benjamin Heymann, Maxime Vono, Patrick Loiseau

机构 * Criteo AI lab（Criteo AI实验室）； FairPlay joint team（FairPlay联合团队）； CREST ； ENSAE ； Institut Polytechnique de Paris（巴黎理工学院）； Inria（法国国家科学与技术研究院）

专题命中偏好对齐：alignment(title,abstract)；DPO(abstract,abstract_cn)；分类 cs.LG

AI总结研究大语言模型对齐中数据估值问题，针对基于Shapley值估值计算量大的挑战，提出顺序偏好优化方法，可高效近似Shapley值，计算真实偏好数据集的Shapley值并揭示各源对模型对齐的作用。

2606.27210 2026-06-26 cs.CL 新提交 85%

Paved with True Intents: Intent-Aware Training Improves LLM Safety Classification Across Training Regimes

以真实意图铺路：意图感知训练提升跨训练机制下的LLM安全分类

Jeremias Ferrao, Niclas Müller-Hof, Iustin Sîrbu, Traian Rebedea, Yftah Ziser

机构 * University of Groningen（格罗宁根大学）； University Politehnica of Bucharest（布加勒斯特理工大学）； NVIDIA（英伟达）

专题命中偏好对齐：safety(title,abstract)；DPO(abstract,abstract_cn)；分类 cs.CL

AI总结提出意图感知训练方法，通过显式建模用户意图信号提升安全分类器鲁棒性，在多个训练机制下取得最优平均性能。

1. 偏好对齐 3252 篇

Improved Algorithms for Differentially Private Language Model Alignment

Sharpe Ratio-Guided Active Learning for Preference Optimization in RLHF

Collab: Controlled Decoding using Mixture of Agents for LLM Alignment

Sample Efficient Preference Alignment in LLMs via Active Exploration

Bootstrapping Language Models with DPO Implicit Rewards

Cannot or Should Not? Automatic Analysis of Refusal Composition in IFT/RLHF Datasets and Refusal Behavior of Black-Box LLMs

Energy-Based Preference Model Offers Better Offline Alignment than the Bradley-Terry Preference Model

Self-Exploring Language Models: Active Preference Elicitation for Online Alignment

BoNBoN Alignment for Large Language Models and the Sweetness of Best-of-n Sampling

Measuring memorization in RLHF for code completion

SeRA: Self-Reviewing and Alignment of Large Language Models using Implicit Reward Margins

Trustworthy Human-AI Collaboration: Reinforcement Learning with Human Feedback and Physics Knowledge for Safe Autonomous Driving

Fundamental Limitations of Alignment in Large Language Models

Online Merging Optimizers for Boosting Rewards and Mitigating Tax in Alignment

Dialectical Alignment: Resolving the Tension of 3H and Security Threats of LLMs

HRLAIF: Improvements in Helpfulness and Harmlessness in Open-domain Reinforcement Learning From AI Feedback

Human Alignment of Large Language Models through Online Preference Optimisation

Direct Language Model Alignment from Online AI Feedback

The Language Barrier: Dissecting Safety Challenges of LLMs in Multilingual Contexts

Diffusion Model Alignment Using Direct Preference Optimization

TIS-DPO: Token-level Importance Sampling for Direct Preference Optimization With Estimated Weights

The Hitchhiker's Guide to Agentic AI: From Foundations to Systems

Substrate Asymmetry in User-Side Memory: A Diagnostic Framework

MaPPO: Maximum a Posteriori Preference Optimization with Prior Knowledge

Modality-Fair Preference Optimization for Trustworthy MLLM Alignment

Behavioral Reprogramming of Open-Weights Models: Cognitive Plasticity and Alignment Bounds

Evaluating Risks in Weak-to-Strong Alignment: A Bias-Variance Perspective

Exploring Post-Training Alignment of Small Language Models for Biomedical Data-to-Text Generation: A Case Study of Medication Leaflet

Shapley-based Data Valuation for LLM Alignment via Sequential Preference Optimization

Paved with True Intents: Intent-Aware Training Improves LLM Safety Classification Across Training Regimes