大模型对齐与安全 - arXivDaily 专题

2512.05464 2025-12-08 cs.CL cs.AI 84%

Dynamic Alignment for Collective Agency: Toward a Scalable Self-Improving Framework for Open-Ended LLM Alignment

动态对齐与集体代理：迈向一个可扩展的自我改进框架以实现开放式的LLM对齐

Panatchakorn Anantaprayoon, Nataliia Babina, Jad Tarifi, Nima Asgharbeygi

机构 * Integral AI

专题命中偏好对齐：alignment(title,abstract)；harmlessness(abstract)；分类 cs.CL、cs.AI

AI总结本文提出动态对齐框架，通过集体代理价值实现LLM的自我改进和可扩展对齐。

Comments 8 pages, 4 figures, to appear in AAAI 2026 AIGOV Workshop

2511.18084 2025-11-25 cs.LG cs.AI 84%

The Alignment Paradox of Medical Large Language Models in Infertility Care: Decoupling Algorithmic Improvement from Clinical Decision-making Quality

医学大语言模型在不孕症护理中的对齐悖论：解耦算法改进与临床决策质量

Dou Liu, Ying Long, Sophia Zuoqiu, Kaipeng Xie, Runze Yang, Di Liu, Kang Li, Yiting Lin, Hanyi Liu, Rong Yin, Tian Tang

机构 * Department of Obstetrics and Gynecology, West China Second University Hospital（妇产科部门，西昌第二大学医院）； Reproductive Medical Center, Department of Obstetrics and Gynecology, West China Second University Hospital（生殖医学中心，妇产科部门，西昌第二大学医院）

专题命中偏好对齐：alignment(title,abstract)；DPO(abstract)；分类 cs.AI、cs.LG

AI总结本研究发现，尽管GRPO在算法准确性上表现最佳，但临床医生更偏好SFT模型，因其推理更清晰且治疗可行性更高，揭示了算法改进与临床信任之间的对齐悖论。

Comments 22 pages 5 figures

2511.16324 2025-11-21 cs.CL cs.AI 84%

SDA: Steering-Driven Distribution Alignment for Open LLMs without Fine-Tuning

SDA：无微调的开放语言模型分布对齐框架

Wei Xia, Zhi-Hong Deng

专题命中偏好对齐：alignment(title,abstract)；harmlessness(abstract)；分类 cs.CL、cs.AI

AI总结 SDA 提出了一种无需微调的开源 LLMs 分布对齐框架，通过用户指令动态调整输出概率，提升模型与人类意图的一致性。

2507.20964 2025-11-20 cs.AI cs.CC cs.GT cs.LG cs.MA 84%

Core Safety Values for Provably Corrigible Agents

Aran Nayebi

机构 * Aran Nayebi（独立研究者）

专题命中偏好对齐：safety(title,abstract)；RLHF(abstract)；分类 cs.AI、cs.LG

Comments 14 pages. To appear in AAAI 2026 Machine Ethics Workshop (W37) Proceedings

2505.23349 2025-11-12 cs.LG cs.AI 84%

Towards Reward Fairness in RLHF: From a Resource Allocation Perspective

Sheng Ouyang, Yulan Hu, Ge Chen, Qingyang Li, Fuzheng Zhang, Yong Liu

机构 * Gaoling School of Artificial Intelligence, Renmin University of China（中国人民大学人工智能学院）； Beijing Key Laboratory of Research on Large Models and Intelligent Governance（北京大型模型与智能治理研究重点实验室）； Engineering Research Center of Next-Generation Intelligent Search and Recommendation, MOE（下一代智能搜索与推荐工程技术研究中心，教育部）； Kuaishou Technology（快手科技）； University of Chinese Academy of Sciences（中国科学院大学）

专题命中偏好对齐：RLHF(title,abstract)；alignment(abstract)；分类 cs.AI、cs.LG

Comments Accepted to ACL 2025

2508.18312 2025-11-10 cs.LG cs.AI 84%

What Matters in Data for DPO?

Yu Pan, Zhongze Cai, Guanting Chen, Huaiyang Zhong, Chonghuan Wang

机构 * University of Sydney（悉尼大学）； Imperial College London（伦敦帝国理工学院）； University of North Carolina at Chapel Hill（北卡罗来纳大学教堂山分校）； Virginia Tech（弗吉尼亚理工大学）； University of Texas at Dallas（德克萨斯大学达拉斯分校）

专题命中偏好对齐：DPO(title,abstract)；alignment(abstract)；分类 cs.AI、cs.LG

2510.23965 2025-10-30 cs.AI cs.LG stat.ML 84%

The Sign Estimator: LLM Alignment in the Face of Choice Heterogeneity

Ali Aouad, Aymane El Gadarri, Vivek F. Farias

机构 * MIT（麻省理工学院）

专题命中偏好对齐：alignment(title,abstract)；RLHF(abstract)；分类 cs.AI、cs.LG

2405.12961 2025-10-24 cs.LG cs.AI physics.chem-ph q-bio.QM 84%

Aligning Transformers with Continuous Feedback via Energy Rank Alignment

Shriram Chennakesavalu, Frank Hu, Sebastian Ibarraran, Grant M. Rotskoff

机构 * Department of Chemistry Stanford University（化学系斯坦福大学）

专题命中偏好对齐：alignment(title,abstract)；DPO(abstract)；分类 cs.AI、cs.LG

2510.13512 2025-10-16 cs.LG cs.AI 84%

Offline and Online KL-Regularized RLHF under Differential Privacy

Yulian Wu, Rushil Thareja, Praneeth Vepakomma, Francesco Orabona

机构 * King Abdullah University of Science and Technology (KAUST)（卡布尔大学科学与技术学院）； Mohamed bin Zayed University of Artificial Intelligence (MBZUAI)（穆罕默德·本·扎耶德人工智能大学）； Massachusetts Institute of Technology (MIT)（麻省理工学院）

专题命中偏好对齐：RLHF(title,abstract)；alignment(abstract)；分类 cs.AI、cs.LG

2510.12044 2025-10-15 cs.CL cs.AI 84%

Hierarchical Alignment: Surgical Fine-Tuning via Functional Layer Specialization in Large Language Models

Yukun Zhang, Qi Dong

机构 * The Chinese University of Hong Kong（香港中文大学）； Fudan University（复旦大学）

专题命中偏好对齐：alignment(title,abstract)；DPO(abstract)；分类 cs.CL、cs.AI

2510.05410 2025-10-08 cs.CL cs.LG 84%

Aligning Language Models with Clinical Expertise: DPO for Heart Failure Nursing Documentation in Critical Care

Junyi Fan, Li Sun, Negin Ashrafi, Kamiar Alaei, Maryam Pishgar

机构 * University of Southern California（南加州大学）； California State University（加州州立大学）

专题命中偏好对齐：DPO(title,abstract)；safety(abstract)；分类 cs.CL、cs.LG

2509.24610 2025-10-01 cs.LG cs.CL 84%

OrthAlign: Orthogonal Subspace Decomposition for Non-Interfering Multi-Objective Alignment

Liang Lin, Zhihao Xu, Junhao Dong, Jian Zhao, Yuchen Yuan, Guibin Zhang, Miao Yu, Yiming Zhang, Zhengtao Yao, Huahui Yi, Dongrui Liu, Xinfeng Li, Kun Wang

机构 * Institute of Artificial Intelligence (TeleAI), China Telecom（人工智能研究院（TeleAI），中国电信）； RUC（中国人民大学）； USTC（University of Science and Technology of China）； NTU（National University of Technology）； NUS（National University of Singapore）； USC（University of Southern California）； SCU（Sichuan University）； Shanghai Artificial Intelligence Laboratory（上海人工智能实验室）

专题命中偏好对齐：alignment(title,abstract)；harmlessness(abstract)；分类 cs.CL、cs.LG

2502.00666 2025-09-29 cs.LG cs.AI stat.ML 84%

Avoiding $\mathbf{exp(R_{max})}$ scaling in RLHF through Preference-based Exploration

Mingyu Chen, Yiding Chen, Wen Sun, Xuezhou Zhang

机构 * Department of Electrical & Computer Engineering（电气与计算机工程系）； Boston University（波士顿大学）； Department of Computer Science（计算机科学系）； Cornell University（康奈尔大学）； Faculty of Computing & Data Sciences（计算与数据科学学院）

专题命中偏好对齐：RLHF(title,abstract)；alignment(abstract)；分类 cs.AI、cs.LG

2508.21016 2025-08-29 cs.LG cs.AI 84%

Inference-Time Alignment Control for Diffusion Models with Reinforcement Learning Guidance

Luozhijie Jin, Zijie Qiu, Jie Liu, Zijie Diao, Lifeng Qiao, Ning Ding, Alex Lamb, Xipeng Qiu

专题命中偏好对齐：alignment(title,abstract)；RLHF(abstract)；分类 cs.AI、cs.LG

2412.19792 2025-08-22 cs.LG cs.CL cs.IT math.IT 84%

InfAlign: Inference-aware language model alignment

Ananth Balashankar, Ziteng Sun, Jonathan Berant, Jacob Eisenstein, Michael Collins, Adrian Hutter, Jong Lee, Chirag Nagpal, Flavien Prost, Aradhana Sinha, Ananda Theertha Suresh, Ahmad Beirami

机构 * Google DeepMind（谷歌DeepMind）； Google Research（谷歌研究）

专题命中偏好对齐：alignment(title,abstract)；RLHF(abstract)；分类 cs.CL、cs.LG

2506.00845 2025-08-19 cs.LG cs.CL 84%

Generalizable LLM Learning of Graph Synthetic Data with Post-training Alignment

Yizhuo Zhang, Heng Wang, Shangbin Feng, Zhaoxuan Tan, Xinyun Liu, Yulia Tsvetkov

机构 * University of Washington（华盛顿大学）； Xi’an Jiaotong University（西安交通大学）； University of Notre Dame（圣母大学）； Google（谷歌）

专题命中偏好对齐：alignment(title,abstract)；DPO(abstract)；分类 cs.CL、cs.LG

Comments 8 pages, 1 figures, 2 tables. Experimental code and results are publicly available at https://anonymous.4open.science/r/Graph_RL-BF08/readme.md

2504.13134 2025-08-06 cs.CL cs.LG stat.ML 84%

Energy-Based Reward Models for Robust Language Model Alignment

Anamika Lochab, Ruqi Zhang

机构 * Department of Computer Science（计算机科学系）； Purdue University（普渡大学）

专题命中偏好对齐：alignment(title,abstract)；safety(abstract)；分类 cs.CL、cs.LG

Comments Accepted by COLM 2025

2406.16306 2025-08-05 cs.CL cs.LG stat.ML 84%

Cascade Reward Sampling for Efficient Decoding-Time Alignment

Bolian Li, Yifan Wang, Anamika Lochab, Ananth Grama, Ruqi Zhang

机构 * Department of Computer Science（计算机科学系）

专题命中偏好对齐：alignment(title,abstract)；safety(abstract)；分类 cs.CL、cs.LG

2502.05773 2025-07-28 cs.LG cs.AI stat.ML 84%

PIPA: Preference Alignment as Prior-Informed Statistical Estimation

Junbo Li, Zhangyang Wang, Qiang Liu

机构 * The University of Texas at Austin, US（德克萨斯大学奥斯汀分校）

专题命中偏好对齐：alignment(title,abstract)；DPO(abstract)；分类 cs.AI、cs.LG

2502.03699 2025-07-25 cs.CL cs.AI cs.IR 84%

LLM Alignment as Retriever Optimization: An Information Retrieval Perspective

Bowen Jin, Jinsung Yoon, Zhen Qin, Ziqi Wang, Wei Xiong, Yu Meng, Jiawei Han, Sercan O. Arik

机构 * University of Illinois at Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）； Google Cloud AI Research（谷歌云人工智能研究）； Google DeepMind（谷歌DeepMind）； University of Virginia（弗吉尼亚大学）

专题命中偏好对齐：alignment(title,abstract)；trustworthy(abstract)；分类 cs.CL、cs.AI

Comments 26 pages

2502.18699 2025-07-23 cs.CL cs.LG stat.ME 84%

MPO: An Efficient Post-Processing Framework for Mixing Diverse Preference Alignment

Tianze Wang, Dongnan Gui, Yifan Hu, Shuhang Lin, Linjun Zhang

机构 * Department of Statistics, Rutgers University, New Brunswick, United States ； Department of Computer Science, Rutgers University, New Brunswick, United States ； College of Management of Technology, EPFL, Switzerland ； Department of Computer Science, ETH Zurich, Switzerland

专题命中偏好对齐：alignment(title,abstract)；RLHF(abstract)；分类 cs.CL、cs.LG

Comments ICML 2025

2507.07725 2025-07-11 cs.CL cs.AI 84%

Not All Preferences are What You Need for Post-Training: Selective Alignment Strategy for Preference Optimization

Zhijin Dong

机构 * Peking University（北京大学）

专题命中偏好对齐：alignment(title,abstract)；DPO(abstract)；分类 cs.CL、cs.AI

2410.08316 2025-06-23 cs.LG cs.CL math.OC 84%

COS-DPO: Conditioned One-Shot Multi-Objective Fine-Tuning Framework

Yinuo Ren, Tesi Xiao, Michael Shavlovsky, Lexing Ying, Holakou Rahmanian

机构 * Institute for Computational and Mathematical Engineering (ICME), Stanford University（计算与数学工程研究所（ICME），斯坦福大学）； Amazon（亚马逊）； Department of Mathematics, Stanford University（数学系，斯坦福大学）

专题命中偏好对齐：DPO(title,abstract)；alignment(abstract)；分类 cs.CL、cs.LG

Comments Published at UAI 2025

2411.06424 2025-06-10 cs.LG cs.CL 84%

How Does DPO Reduce Toxicity? A Mechanistic Neuron-Level Analysis

Yushi Yang, Filip Sondej, Harry Mayne, Andrew Lee, Adam Mahdi

机构 * University of Oxford（牛津大学）； Jagiellonian University（雅盖隆大学）； Harvard University（哈佛大学）

专题命中偏好对齐：DPO(title,abstract)；safety(abstract)；分类 cs.CL、cs.LG

Journal ref NeurIPS 2024 Workshop on Socially Responsible Language Modelling Research (SoLaR)

2503.04647 2025-06-06 cs.CL cs.AI 84%

Implicit Cross-Lingual Rewarding for Efficient Multilingual Preference Alignment

Wen Yang, Junhong Wu, Chen Wang, Chengqing Zong, Jiajun Zhang

专题命中偏好对齐：alignment(title,abstract)；DPO(abstract)；分类 cs.CL、cs.AI

Comments Camera ready version for ACL 2025 Findings

2505.23729 2025-06-03 cs.CL cs.AI 84%

Bounded Rationality for LLMs: Satisficing Alignment at Inference-Time

Mohamad Chehade, Soumya Suvra Ghosal, Souradip Chakraborty, Avinash Reddy, Dinesh Manocha, Hao Zhu, Amrit Singh Bedi

专题命中偏好对齐：alignment(title,abstract)；harmlessness(abstract)；分类 cs.CL、cs.AI

Comments Accepted at ICML 2025

2505.24369 2025-06-02 cs.LG cs.AI 84%

Adversarial Preference Learning for Robust LLM Alignment

Yuanfu Wang, Pengyu Wang, Chenyang Xi, Bo Tang, Junyi Zhu, Wenqiang Wei, Chen Chen, Chao Yang, Jingfeng Zhang, Chaochao Lu, Yijun Niu, Keming Mao, Zhiyu Li, Feiyu Xiong, Jie Hu, Mingchuan Yang

专题命中偏好对齐：alignment(title)；RLHF(abstract)；harmlessness(abstract)；分类 cs.AI、cs.LG

Comments Accepted at ACL2025 Findings

2501.09620 2025-05-30 cs.LG cs.AI 84%

Beyond Reward Hacking: Causal Rewards for Large Language Model Alignment

Chaoqi Wang, Zhuokai Zhao, Yibo Jiang, Zhaorun Chen, Chen Zhu, Yuxin Chen, Jiayi Liu, Lizhu Zhang, Xiangjun Fan, Hao Ma, Sinong Wang

机构 * Meta ； University of Chicago（芝加哥大学）

专题命中偏好对齐：alignment(title,abstract)；RLHF(abstract)；分类 cs.AI、cs.LG

2505.17558 2025-05-26 cs.CL cs.AI 84%

Teaching with Lies: Curriculum DPO on Synthetic Negatives for Hallucination Detection

Shrey Pandit, Ashwin Vinod, Liu Leqi, Ying Ding

机构 * The University of Texas at Austin（德克萨斯大学奥斯汀分校）

专题命中偏好对齐：DPO(title,abstract)；alignment(abstract)；分类 cs.CL、cs.AI

Comments Code and dataset are available at https://teachingwithlies.github.io/

2503.10669 2025-05-20 cs.CL cs.AI 84%

UC-MOA: Utility-Conditioned Multi-Objective Alignment for Distributional Pareto-Optimality

Zelei Cheng, Xin-Qiang Cai, Yuting Tang, Pushi Zhang, Boming Yang, Masashi Sugiyama, Xinyu Xing

机构 * Northwestern University（西北大学）； RIKEN-AIP（日本理化学研究所-AIP）； The University of Tokyo（东京大学）； Microsoft Research Asia（微软亚洲研究院）

专题命中偏好对齐：alignment(title,abstract)；RLHF(abstract)；分类 cs.CL、cs.AI

Comments Language Modeling, Machine Learning for NLP, Distributional Pareto-Optimal

1. 偏好对齐 3261 篇

Dynamic Alignment for Collective Agency: Toward a Scalable Self-Improving Framework for Open-Ended LLM Alignment

The Alignment Paradox of Medical Large Language Models in Infertility Care: Decoupling Algorithmic Improvement from Clinical Decision-making Quality

SDA: Steering-Driven Distribution Alignment for Open LLMs without Fine-Tuning

Core Safety Values for Provably Corrigible Agents

Towards Reward Fairness in RLHF: From a Resource Allocation Perspective

What Matters in Data for DPO?

The Sign Estimator: LLM Alignment in the Face of Choice Heterogeneity

Aligning Transformers with Continuous Feedback via Energy Rank Alignment

Offline and Online KL-Regularized RLHF under Differential Privacy

Hierarchical Alignment: Surgical Fine-Tuning via Functional Layer Specialization in Large Language Models

Aligning Language Models with Clinical Expertise: DPO for Heart Failure Nursing Documentation in Critical Care

OrthAlign: Orthogonal Subspace Decomposition for Non-Interfering Multi-Objective Alignment

Avoiding $\mathbf{exp(R_{max})}$ scaling in RLHF through Preference-based Exploration

Inference-Time Alignment Control for Diffusion Models with Reinforcement Learning Guidance

InfAlign: Inference-aware language model alignment

Generalizable LLM Learning of Graph Synthetic Data with Post-training Alignment

Energy-Based Reward Models for Robust Language Model Alignment

Cascade Reward Sampling for Efficient Decoding-Time Alignment

PIPA: Preference Alignment as Prior-Informed Statistical Estimation

LLM Alignment as Retriever Optimization: An Information Retrieval Perspective

MPO: An Efficient Post-Processing Framework for Mixing Diverse Preference Alignment

Not All Preferences are What You Need for Post-Training: Selective Alignment Strategy for Preference Optimization

COS-DPO: Conditioned One-Shot Multi-Objective Fine-Tuning Framework

How Does DPO Reduce Toxicity? A Mechanistic Neuron-Level Analysis

Implicit Cross-Lingual Rewarding for Efficient Multilingual Preference Alignment

Bounded Rationality for LLMs: Satisficing Alignment at Inference-Time

Adversarial Preference Learning for Robust LLM Alignment

Beyond Reward Hacking: Causal Rewards for Large Language Model Alignment

Teaching with Lies: Curriculum DPO on Synthetic Negatives for Hallucination Detection

UC-MOA: Utility-Conditioned Multi-Objective Alignment for Distributional Pareto-Optimality