大模型对齐与安全 - arXivDaily 专题

2507.16951 2025-07-24 cs.CL 86%

Harnessing RLHF for Robust Unanswerability Recognition and Trustworthy Response Generation in LLMs

Shuyuan Lin, Lei Duan, Philip Hughes, Yuxuan Sheng

机构 * Sichuan University of Science and Engineering（四川理工学院）； Zagazig University（扎加齐亚大学）

专题命中偏好对齐：RLHF(title,abstract)；trustworthy(title)；分类 cs.CL

2605.11632 2026-06-05 cs.CL cs.AI 86%

Macro: Enhancing Multilingual Counterfactual Explanations through Alignment-as-Preference Optimization

Macro: 通过偏好对齐优化提升多语言反事实解释

Yilong Wang, Qianli Wang, Bohao Chu, Yihong Liu, Jing Yang, Simon Ostermann

机构 * Technische Universität Berlin（柏林技术大学）； German Research Center for Artificial Intelligence (DFKI)（德国人工智能研究中心）； University of Duisburg-Essen（杜伊斯堡- Essen大学）； LMU Munich（慕尼黑大学）； Munich Center for Machine Learning (MCML)（慕尼黑机器学习中心）； Saarland Informatics Campus（萨尔兰州信息学校区）； BIFOLD – Berlin Institute for the Foundations of Learning and Data（柏林学习与数据基础研究院）； Centre for European Research in Trusted AI (CERTAIN)（可信人工智能欧洲研究中心）

专题命中偏好对齐：alignment(title,abstract)；DPO(abstract,abstract_cn)；分类 cs.CL、cs.AI

AI总结本文提出Macro框架，通过直接偏好优化改进多语言反事实解释，提升有效性的同时保持最小性，避免翻译基线的严重最小性问题，并在多个指标上优于监督微调方法。

Comments In submission

2605.23961 2026-05-26 q-bio.BM cs.AI cs.LG 86%

Multimodal Alignment and Preference Optimization for Zero-Shot Conditional RNA Generation

多模态对齐与偏好优化用于零样本条件RNA生成

Roman Klypa, Alberto Bietti, Sergei Grudinin

机构 * Univ. Grenoble Alpes, CNRS, Grenoble INP, LJK（格勒诺布尔阿尔卑斯大学、法国国家科学研究中心、格勒诺布尔INP、LJK实验室）； Center for Computational Mathematics, Flatiron Institute（计算数学中心、Flatiron研究所）

专题命中偏好对齐：alignment(title,abstract)；DPO(abstract,abstract_cn)；分类 cs.AI、cs.LG

AI总结提出Moirain框架，通过多模态监督微调和直接偏好优化实现条件RNA序列生成，在零样本条件下生成具有高结合亲和力的生物合理RNA序列。

2605.17342 2026-05-19 cs.CL cs.AI 86%

Transitivity Meets Cyclicity: Explicit Preference Decomposition for Dynamic Large Language Model Alignment

传递性与循环性：动态大语言模型对齐的显式偏好分解

Yucong Huang, Xiucheng Li, Kaiqi Zhao, Jing Li

机构 * Harbin Institute of Technology, Shenzhen, China（哈尔滨工业大学深圳校区）

专题命中偏好对齐：alignment(title,abstract)；RLHF(abstract,abstract_cn)；分类 cs.CL、cs.AI

AI总结本文提出Hybrid Reward-Cyclic模型，通过博弈论分解显式分离传递性和循环性偏好，结合动态自我博弈优化方法提升大语言模型对齐效果，实验证明其在混合传递-循环设置中具有结构优势和更高的准确率。

Comments Accepted by ICML 2026

2605.06375 2026-05-12 cs.LG cs.AI math.ST stat.TH 86%

A Unified Pair-GRPO Family: From Implicit to Explicit Preference Constraints for Stable and General RL Alignment

一对GRPO家族的统一：从隐式到显式的偏好约束用于稳定和通用的RL对齐

Hao Yu

机构 * Department of Automation, Tsinghua University（清华大学自动化系）

专题命中偏好对齐：alignment(title,abstract)；RLHF(abstract,abstract_cn)；分类 cs.AI、cs.LG

AI总结本文提出了一种统一的偏好基于RL优化框架，通过Soft-Pair-GRPO和Hard-Pair-GRPO两种变体，解决了主流配对偏好学习中政策更新不稳定、梯度方向模糊等问题，提升了对齐质量与训练稳定性。

2604.02652 2026-04-06 cs.LG cs.AI 86%

Generalization Limits of Reinforcement Learning Alignment

强化学习对齐的泛化极限

Haruhi Shida, Koo Imai, Keigo Kansa

机构 * Aladdin Security Inc（阿拉丁安全公司）

专题命中偏好对齐：alignment(title,abstract)；RLHF(abstract)；safety(abstract)；分类 cs.AI、cs.LG

AI总结本文研究了强化学习对齐技术的泛化能力限制，提出复合越狱方法，通过结合多种攻击技术提高攻击成功率，验证了安全训练无法广泛泛化 hypothesis。

Comments 7 pages, 2 figures, 2 tables, accepted at JSAI 2026

2603.19741 2026-03-23 cs.LG cs.CL 86%

FedPDPO: Federated Personalized Direct Preference Optimization for Large Language Model Alignment

FedPDPO：联邦个性化直接偏好优化用于大语言模型对齐

Kewen Zhu, Liping Yi, Zhiming Zhao, Zhuang Qi, Han Yu, Qinghua Hu

机构 * College of Intelligence and Computing, Tianjin University, Tianjin, China（智能与计算学院，天津大学，天津，中国）； College of Computing and Data Science, Nanyang Technological University, Singapore（计算与数据科学学院，南洋理工大学，新加坡）

专题命中偏好对齐：alignment(title,abstract)；RLHF(abstract)；DPO(abstract)；分类 cs.CL、cs.LG

AI总结本文提出FedPDPO，一种联邦个性化直接偏好优化框架，通过参数高效微调和个性化奖励头解决非iid数据下的大语言模型对齐问题，实验显示在联邦内域和跨域设置中平均准确率提升达4.80%。

Comments under review

2505.20065 2026-03-05 cs.LG cs.AI 86%

SafeDPO: A Simple Approach to Direct Preference Optimization with Enhanced Safety

SafeDPO: 一种简单的方法用于直接偏好优化并增强安全性

Geon-Hyeong Kim, Yu Jin Kim, Byoungjip Kim, Honglak Lee, Kyunghoon Bae, Youngsoo Jang, Moontae Lee

机构 * LG AI Research（LG人工智能研究）

专题命中偏好对齐：safety(title,abstract)；alignment(abstract)；RLHF(abstract)；分类 cs.AI、cs.LG

AI总结 SafeDPO通过简单理论驱动的目标，实现轻量级且有效的安全对齐，提升安全性的同时保持有用性。

Comments 40 pages

Journal ref In Proceedings of the International Conference on Learning Representations (ICLR), 2026

2602.16053 2026-02-20 cs.LG cs.CL 86%

Multi-Objective Alignment of Language Models for Personalized Psychotherapy

多目标对齐语言模型以实现个性化心理治疗

Mehrab Beikzadeh, Yasaman Asadollah Salmanpour, Ashima Suvarna, Sriram Sankararaman, Matteo Malgaroli, Majid Sarrafzadeh, Saadia Gabriel

机构 * Department of Computer Science, University of California, Los Angeles, CA, USA（计算机科学系，加州大学洛杉矶分校）； Department of Psychology, The University of Texas at Austin, Austin, TX, USA（心理学系，德克萨斯大学奥斯汀分校）； Department of Psychiatry, NYU Grossman School of Medicine, New York, NY, USA（精神病学系，纽约大学格罗斯曼医学院）

专题命中偏好对齐：alignment(title,abstract)；DPO(abstract)；safety(abstract)；分类 cs.CL、cs.LG

AI总结本文提出多目标对齐框架，通过直接偏好优化提升语言模型在心理治疗中的共情与安全性能，实验显示其在平衡患者偏好与临床安全方面优于传统方法。

2507.00665 2026-02-02 cs.CL cs.AI 86%

SAFER: Probing Safety in Reward Models with Sparse Autoencoder

SAFER: 通过稀疏自动编码器探索奖励模型的安全性

Wei Shi, Ziyuan Xie, Sihang Li, Xiang Wang

机构 * University of Science and Technology of China（中国科学技术大学）

专题命中偏好对齐：safety(title,abstract)；alignment(abstract)；RLHF(abstract)；分类 cs.CL、cs.AI

AI总结 SAFER通过稀疏自动编码器探索奖励模型的安全性，揭示可解释特征并改进安全性对齐。

2601.17329 2026-01-27 cs.LG cs.AI 86%

Conformal Feedback Alignment: Quantifying Answer-Level Reliability for Robust LLM Alignment

符合反馈对齐：量化答案级可靠性以实现鲁棒的大语言模型对齐

Tiejin Chen, Xiaoou Liu, Vishnu Nandam, Kuan-Ru Liou, Hua Wei

机构 * Arizona State University（亚利桑那州立大学）

专题命中偏好对齐：alignment(title,abstract)；RLHF(abstract)；DPO(abstract)；分类 cs.AI、cs.LG

AI总结 Conformal Feedback Alignment通过量化答案级可靠性提升大语言模型对齐的鲁棒性和数据效率。

Comments Accetped to Findings of EACL

2510.06092 2026-01-21 cs.LG cs.CL 86%

Learning from Failures: Understanding LLM Alignment through Failure-Aware Inverse RL

从失败中学习：通过失败意识反向强化学习理解LLM对齐

Nyal Patel, Matthieu Bou, Arjun Jagota, Satyapriya Krishna, Sonali Parbhoo

机构 * Imperial College London（帝国理工学院伦敦分校）； Amazon AGI（亚马逊人工智能实验室）

专题命中偏好对齐：alignment(title,abstract)；RLHF(abstract)；safety(abstract)；分类 cs.CL、cs.LG

AI总结本文提出一种失败意识反向强化学习算法，通过聚焦于误分类或困难的例子来提取更准确的奖励函数，从而提升LLM对齐的可解释性和安全性。

Comments Preprint

2601.06157 2026-01-13 cs.LG cs.AI 86%

ECLIPTICA -- A Framework for Switchable LLM Alignment via CITA - Contrastive Instruction-Tuned Alignment

ECLIPTICA -- 一种通过CITA进行可切换LLM对齐的框架

Kapil Wanaskar, Gaytri Jena, Vinija Jain, Aman Chadha, Amitava Das

机构 * San José State University（圣何塞州立大学）； Google（谷歌）； Apple（苹果）； Pragya Lab, BITS Pilani Goa, India（BITS Pilani Goa 印度分校实验室）

专题命中偏好对齐：alignment(title,abstract)；DPO(abstract)；safety(abstract)；分类 cs.AI、cs.LG

AI总结 ECLIPTICA通过CITA实现LLM对齐的可切换框架，以高效率提升指令对齐性能。

2409.00162 2025-12-25 cs.CL cs.AI 86%

Sequence to Sequence Reward Modeling: Improving RLHF by Language Feedback

序列到序列奖励建模：通过语言反馈改进RLHF

Jiayi Zhou, Jiaming Ji, Juntao Dai, Dong Li, Yaodong Yang

专题命中偏好对齐：RLHF(title,abstract)；alignment(abstract)；safety(abstract)；分类 cs.CL、cs.AI

AI总结通过序列到序列奖励建模方法改进RLHF，提升LLMs对齐人类意图的性能。

Comments 7 pages

2512.13240 2025-12-16 cs.AI cs.LG 86%

Reflective Preference Optimization (RPO): Enhancing On-Policy Alignment via Hint-Guided Reflection

反射偏好优化（RPO）：通过提示引导的反思增强策略对齐

Zihui Zhao, Zechang Li

机构 * Tsinghua Shenzhen International Graduate School, Tsinghua University（清华大学深圳国际研究生院，清华大学）； Alibaba Group（阿里巴巴集团）

专题命中偏好对齐：alignment(title,abstract)；RLHF(abstract)；DPO(abstract)；分类 cs.AI、cs.LG

AI总结 RPO通过提示引导的反思增强策略对齐，减少幻觉并提升多模态基准性能。

2508.06783 2025-12-11 cs.LG cs.AI cs.CR cs.IT math.IT 86%

PROPS: Progressively Private Self-alignment of Large Language Models

PROPS: 大型语言模型的逐步隐私自对齐

Noel Teku, Fengwei Tian, Payel Bhattacharjee, Souradip Chakraborty, Amrit Singh Bedi, Ravi Tandon

专题命中偏好对齐：alignment(title,abstract)；RLHF(abstract)；DPO(abstract)；分类 cs.AI、cs.LG

AI总结 PROPS通过多阶段隐私保护对齐框架，在保护偏好标签隐私的同时提升LLM对齐效果，实现更高的胜率。

Comments Accepted in the Transactions on Machine Learning Research (TMLR), 2025

Journal ref Transactions on ML Research (TMLR) 2025

2511.21050 2025-11-27 cs.LG cs.AI stat.ML 86%

Breaking the Safety-Capability Tradeoff: Reinforcement Learning with Verifiable Rewards Maintains Safety Guardrails in LLMs

打破安全性与能力的权衡：具有可验证奖励的强化学习在LLMs中维持安全护栏

Dongkyu Derek Cho, Huan Song, Arijit Ghosh Chowdhury, Haotian An, Yawei Wang, Rohit Thekkanal, Negin Sokhandan, Sharlina Keshava, Hannah Marlowe

专题命中偏好对齐：safety(title,abstract)；alignment(abstract)；RLHF(abstract)；分类 cs.AI、cs.LG

AI总结本文提出通过可验证奖励的强化学习方法，在提升LLM推理能力的同时维持安全性，挑战了传统安全性与能力权衡的假设。

Comments AAAI-26 Workshop on Post-AI Formal Methods

2505.17859 2025-10-27 cs.LG cs.AI stat.ML 86%

Scalable Valuation of Human Feedback through Provably Robust Model Alignment

Masahiro Fujisawa, Masaki Adachi, Michael A. Osborne

机构 * The University of Osaka（大阪大学）； Lattice Lab, Toyota Motor Corporation（丰田公司Lattice实验室）； Machine Learning Research Group, University of Oxford（牛津大学机器学习研究组）； RIKEN AIP（理化学研究所AIP）

专题命中偏好对齐：alignment(title,abstract)；RLHF(abstract)；DPO(abstract)；分类 cs.AI、cs.LG

Comments Accepted by the 39th Conference on Neural Information Processing Systems (NeurIPS2025), 49 pages, 7 figures

2510.16167 2025-10-21 cs.LG cs.CL 86%

Alignment is Localized: A Causal Probe into Preference Layers

Archie Chaudhury

机构 * Independent（独立研究者）

专题命中偏好对齐：alignment(title,abstract)；RLHF(abstract)；safety(abstract)；分类 cs.CL、cs.LG

2506.22578 2025-10-14 cs.LG cs.AI stat.ML 86%

The Hidden Link Between RLHF and Contrastive Learning

Xufei Lv, Kehai Chen, Haoyuan Sun, Xuefeng Bai, Min Zhang, Houde Liu, Kehai Chen

机构 * Tsinghua University（清华大学）； Harbin Institute of Technology（哈尔滨工业大学）； Soochow University（苏州大学）

专题命中偏好对齐：RLHF(title,abstract)；alignment(abstract)；DPO(abstract)；分类 cs.AI、cs.LG

2508.07137 2025-09-26 cs.LG cs.AI 86%

A Principled Loss Function for Direct Language Model Alignment

Yuandong Tan

专题命中偏好对齐：alignment(title,abstract)；RLHF(abstract)；DPO(abstract)；分类 cs.AI、cs.LG

2505.19743 2025-08-19 cs.CL cs.LG 86%

Token-level Accept or Reject: A Micro Alignment Approach for Large Language Models

Yang Zhang, Yu Yu, Bo Tang, Yu Zhu, Chuxiong Sun, Wenqiang Wei, Jie Hu, Zipeng Xie, Zhiyu Li, Feiyu Xiong, Edward Chung

机构 * Hong Kong Polytechnic University（香港理工大学）； MemTensor (Shanghai) Technology Co., Ltd（MemTensor（上海）科技有限公司）； University of Science and Technology of China（中国科学技术大学）； China Telecom Corporation Limited Beijing Research Institute（中国电信北京研究院）； Nanjing University of Information Science and Technology（南京信息工程大学）

专题命中偏好对齐：alignment(title,abstract)；RLHF(abstract)；DPO(abstract)；分类 cs.CL、cs.LG

Comments Accepted to 34th International Joint Conference on Artificial Intelligence (IJCAI 2025)

2508.08509 2025-08-13 cs.CL cs.AI 86%

Steerable Pluralism: Pluralistic Alignment via Few-Shot Comparative Regression

Jadie Adams, Brian Hu, Emily Veenhuis, David Joy, Bharadwaj Ravichandran, Aaron Bray, Anthony Hoogs, Arslan Basharat

专题命中偏好对齐：alignment(title,abstract)；RLHF(abstract)；harmlessness(abstract)；分类 cs.CL、cs.AI

Comments AIES '25: Proceedings of the 2025 AAAI/ACM Conference on AI, Ethics, and Society

2507.09406 2025-07-15 cs.LG cs.AI 86%

Adversarial Activation Patching: A Framework for Detecting and Mitigating Emergent Deception in Safety-Aligned Transformers

Santhosh Kumar Ravindran

机构 * Microsoft Corporation（微软公司）

专题命中偏好对齐：safety(title,abstract)；RLHF(abstract)；AI safety(abstract)；分类 cs.AI、cs.LG

2411.07618 2025-07-11 cs.AI cs.CL 86%

Constrain Alignment with Sparse Autoencoders

Qingyu Yin, Chak Tou Leong, Minjun Zhu, Hanqi Yan, Qiang Zhang, Yulan He, Wenjie Li, Jun Wang, Yue Zhang, Linyi Yang

机构 * Zhejiang University（浙江大学）； Hongkong Polytechnic University（香港理工大学）； University College London（伦敦大学学院）； Westlake University（西湖大学）； King's College London（国王学院）； Southern University of Science and Technology（南方科技大学）

专题命中偏好对齐：alignment(title,abstract)；RLHF(abstract)；DPO(abstract)；分类 cs.CL、cs.AI

Journal ref ICML 2025

2502.17507 2025-06-17 cs.LG cs.AI 86%

C2-DPO: Constrained Controlled Direct Preference Optimization

Kavosh Asadi, Julien Han, Idan Pipano, Xingzi Xu, Dominique Perrault-Joncas, Shoham Sabach, Karim Bouyarmane, Mohammad Ghavamzadeh

机构 * Amazon（亚马逊公司）； Technion（技术学院）； Duke（杜克大学）

专题命中偏好对齐：DPO(title,abstract)；alignment(abstract)；RLHF(abstract)；分类 cs.AI、cs.LG

2406.12091 2025-06-10 cs.LG cs.CL cs.CR 86%

Is poisoning a real threat to LLM alignment? Maybe more so than you think

Pankayaraj Pathmanathan, Souradip Chakraborty, Xiangyu Liu, Yongyuan Liang, Furong Huang

机构 * University of Maryland（马里兰大学）； Capital One（Capital One公司）

专题命中偏好对齐：alignment(title,abstract)；RLHF(abstract)；DPO(abstract)；分类 cs.CL、cs.LG

Journal ref AAAI 2025

2409.17169 2025-06-05 cs.CL cs.AI 86%

REAL: Response Embedding-based Alignment for LLMs

Honggen Zhang, Xufeng Zhao, Igor Molybog, June Zhang

专题命中偏好对齐：alignment(title,abstract)；RLHF(abstract)；DPO(abstract)；分类 cs.CL、cs.AI

2409.11704 2025-05-26 cs.CL cs.LG 86%

From Lists to Emojis: How Format Bias Affects Model Alignment

Xuanchang Zhang, Wei Xiong, Lichang Chen, Tianyi Zhou, Heng Huang, Tong Zhang

专题命中偏好对齐：alignment(title,abstract)；RLHF(abstract)；DPO(abstract)；分类 cs.CL、cs.LG

Comments Working in progress

2503.17682 2025-05-23 cs.LG cs.AI 86%

Safe RLHF-V: Safe Reinforcement Learning from Multi-modal Human Feedback

Jiaming Ji, Xinyu Chen, Rui Pan, Conghui Zhang, Han Zhu, Jiahao Li, Donghai Hong, Boyuan Chen, Jiayi Zhou, Kaile Wang, Juntao Dai, Chi-Min Chan, Yida Tang, Sirui Han, Yike Guo, Yaodong Yang

机构 * Peking University（北京大学）； Hong Kong University of Science and Technology（香港科技大学）

专题命中偏好对齐：RLHF(title,abstract)；alignment(abstract)；safety(abstract)；分类 cs.AI、cs.LG

1. 偏好对齐 3252 篇

Harnessing RLHF for Robust Unanswerability Recognition and Trustworthy Response Generation in LLMs

Macro: Enhancing Multilingual Counterfactual Explanations through Alignment-as-Preference Optimization

Multimodal Alignment and Preference Optimization for Zero-Shot Conditional RNA Generation

Transitivity Meets Cyclicity: Explicit Preference Decomposition for Dynamic Large Language Model Alignment

A Unified Pair-GRPO Family: From Implicit to Explicit Preference Constraints for Stable and General RL Alignment

Generalization Limits of Reinforcement Learning Alignment

FedPDPO: Federated Personalized Direct Preference Optimization for Large Language Model Alignment

SafeDPO: A Simple Approach to Direct Preference Optimization with Enhanced Safety

Multi-Objective Alignment of Language Models for Personalized Psychotherapy

SAFER: Probing Safety in Reward Models with Sparse Autoencoder

Conformal Feedback Alignment: Quantifying Answer-Level Reliability for Robust LLM Alignment

Learning from Failures: Understanding LLM Alignment through Failure-Aware Inverse RL

ECLIPTICA -- A Framework for Switchable LLM Alignment via CITA - Contrastive Instruction-Tuned Alignment

Sequence to Sequence Reward Modeling: Improving RLHF by Language Feedback

Reflective Preference Optimization (RPO): Enhancing On-Policy Alignment via Hint-Guided Reflection

PROPS: Progressively Private Self-alignment of Large Language Models

Breaking the Safety-Capability Tradeoff: Reinforcement Learning with Verifiable Rewards Maintains Safety Guardrails in LLMs

Scalable Valuation of Human Feedback through Provably Robust Model Alignment

Alignment is Localized: A Causal Probe into Preference Layers

The Hidden Link Between RLHF and Contrastive Learning

A Principled Loss Function for Direct Language Model Alignment

Token-level Accept or Reject: A Micro Alignment Approach for Large Language Models

Steerable Pluralism: Pluralistic Alignment via Few-Shot Comparative Regression

Adversarial Activation Patching: A Framework for Detecting and Mitigating Emergent Deception in Safety-Aligned Transformers

Constrain Alignment with Sparse Autoencoders

C2-DPO: Constrained Controlled Direct Preference Optimization

Is poisoning a real threat to LLM alignment? Maybe more so than you think

REAL: Response Embedding-based Alignment for LLMs

From Lists to Emojis: How Format Bias Affects Model Alignment

Safe RLHF-V: Safe Reinforcement Learning from Multi-modal Human Feedback