大模型对齐与安全 - arXivDaily 专题

2601.06238 2026-01-13 cs.LG cs.AI cs.CL 87%

SPINAL -- Scaling-law and Preference Integration in Neural Alignment Layers

SPINAL -- 神经对齐层中的缩放律与偏好整合

Arion Das, Partha Pratim Saha, Amit Dhanda, Vinija Jain, Aman Chadha, Amitava Das

专题命中偏好对齐：alignment(title,abstract)；RLHF(abstract)；DPO(abstract)；分类 cs.CL、cs.AI、cs.LG

AI总结 SPINAL通过分析神经对齐层的几何特性，量化对齐在深度层的集中表现及稳定性变化。

2510.08256 2025-10-10 cs.LG cs.AI cs.CL 87%

Mix- and MoE-DPO: A Variational Inference Approach to Direct Preference Optimization

Jason Bohne, Pawel Polak, David Rosenberg, Brian Bloniarz, Gary Kazantsev

机构 * Department of Applied Mathematics and Statistics（应用数学与统计学系）； Stony Brook University（石溪大学）； Institute for Advanced Computational Science（先进计算科学研究所）； Center of Excellence in Wireless and Information Technology (CEWIT)（无线与信息技术卓越中心）； AI Innovation Institute（人工智能创新研究院）； Bloomberg（彭博）； Toronto, ON M5J 2S1（多伦多，ON M5J 2S1）； San Francisco, CA 94105（旧金山，CA 94105）； Bloomberg New York, NY 10022（纽约，NY 10022）

专题命中偏好对齐：DPO(title,abstract)；alignment(abstract)；RLHF(abstract)；分类 cs.CL、cs.AI、cs.LG

2410.16033 2025-09-04 cs.CL cs.AI cs.LG 87%

TreeBoN: Enhancing Inference-Time Alignment with Speculative Tree-Search and Best-of-N Sampling

Jiahao Qiu, Yifu Lu, Yifan Zeng, Jiacheng Guo, Jiayi Geng, Chenhao Zhu, Xinzhe Juan, Ling Yang, Huazheng Wang, Kaixuan Huang, Yue Wu, Mengdi Wang

专题命中偏好对齐：alignment(title,abstract)；RLHF(abstract)；DPO(abstract)；分类 cs.CL、cs.AI、cs.LG

2503.02832 2025-07-24 cs.CL cs.AI cs.LG 87%

AlignDistil: Token-Level Language Model Alignment as Adaptive Policy Distillation

Songming Zhang, Xue Zhang, Tong Zhang, Bojie Hu, Yufeng Chen, Jinan Xu

机构 * Key Laboratory of Big Data & Artificial Intelligence in Transportation, (Beijing Jiaotong University), Ministry of Education（大数据与人工智能交通运输联合实验室，（北京交通大学）教育部）； School of Computer Science and Technology, Beijing Jiaotong University, Beijing, China（计算机科学与技术学院，北京交通大学，北京，中国）； Tencent Inc, China（腾讯公司，中国）

专题命中偏好对齐：alignment(title,abstract)；RLHF(abstract)；DPO(abstract)；分类 cs.CL、cs.AI、cs.LG

Comments ACL 2025 Main Conference, code available at: https://github.com/songmzhang/AlignDistil

2501.08617 2025-06-11 cs.LG cs.AI cs.CL 87%

RLHS: Mitigating Misalignment in RLHF with Hindsight Simulation

Kaiqu Liang, Haimin Hu, Ryan Liu, Thomas L. Griffiths, Jaime Fernández Fisac

机构 * Princeton University（普林斯顿大学）

专题命中偏好对齐：RLHF(title,abstract)；alignment(abstract)；DPO(abstract)；分类 cs.CL、cs.AI、cs.LG

Comments 27 pages, 18 figures

2506.02726 2025-06-04 cs.CL cs.AI cs.LG 87%

RACE-Align: Retrieval-Augmented and Chain-of-Thought Enhanced Preference Alignment for Large Language Models

Qihang Yan, Xinyu Zhang, Luming Guo, Qi Zhang, Feifan Liu

机构 * ShanghaiTech University（上海科技大学）； Henan University（河南大学）； Liaoning University of Traditional Chinese Medicine（辽宁中医药大学）

专题命中偏好对齐：alignment(title,abstract)；RLHF(abstract)；DPO(abstract)；分类 cs.CL、cs.AI、cs.LG

2502.12678 2025-05-27 cs.LG cs.AI cs.CL 87%

Multi-Step Alignment as Markov Games: An Optimistic Online Gradient Descent Approach with Convergence Guarantees

Yongtao Wu, Luca Viano, Yihang Chen, Zhenyu Zhu, Kimon Antonakopoulos, Quanquan Gu, Volkan Cevher

专题命中偏好对齐：alignment(title,abstract)；RLHF(abstract)；DPO(abstract)；分类 cs.CL、cs.AI、cs.LG

2410.01639 2025-05-13 cs.LG cs.AI cs.CY 87%

Moral Alignment for LLM Agents

Elizaveta Tennant, Stephen Hailes, Mirco Musolesi

机构 * University College London（伦敦大学学院）； University of Bologna（博洛尼亚大学）

专题命中偏好对齐：alignment(title,abstract)；RLHF(abstract)；DPO(abstract)；分类 cs.AI、cs.CY、cs.LG

Comments Published at the 13th International Conference on Learning Representations (ICLR'25), Singapore, Apr 2025. https://openreview.net/forum?id=MeGDmZjUXy

2410.13828 2025-04-23 cs.LG cs.AI cs.CL 87%

A Common Pitfall of Margin-based Language Model Alignment: Gradient Entanglement

Hui Yuan, Yifan Zeng, Yue Wu, Huazheng Wang, Mengdi Wang, Liu Leqi

机构 * The University of Texas at Austin（德克萨斯大学奥斯汀分校）； Princeton Language & Intelligence（普林斯顿语言与智能）

专题命中偏好对齐：alignment(title,abstract)；RLHF(abstract)；safety(abstract)；分类 cs.CL、cs.AI、cs.LG

2408.15313 2025-04-09 cs.AI cs.CL cs.LG 87%

Bi-Factorial Preference Optimization: Balancing Safety-Helpfulness in Language Models

Wenxuan Zhang, Philip H. S. Torr, Mohamed Elhoseiny, Adel Bibi

专题命中偏好对齐：safety(title,abstract)；RLHF(abstract)；harmlessness(abstract)；分类 cs.CL、cs.AI、cs.LG

Comments The paper has been accepted in ICLR 2025 as spotlight presentation

2410.19933 2025-02-28 cs.LG cs.AI cs.CY 87%

Enhancing Safety in Reinforcement Learning with Human Feedback via Rectified Policy Optimization

Xiyue Peng, Hengquan Guo, Jiawei Zhang, Dongqing Zou, Ziyu Shao, Honghao Wei, Xin Liu

专题命中偏好对齐：safety(title,abstract)；alignment(abstract)；harmlessness(abstract)；分类 cs.AI、cs.CY、cs.LG

2407.13399 2025-02-19 cs.AI cs.CL cs.LG 87%

Correcting the Mythos of KL-Regularization: Direct Alignment without Overoptimization via Chi-Squared Preference Optimization

Audrey Huang, Wenhao Zhan, Tengyang Xie, Jason D. Lee, Wen Sun, Akshay Krishnamurthy, Dylan J. Foster

专题命中偏好对齐：alignment(title,abstract)；RLHF(abstract)；DPO(abstract)；分类 cs.CL、cs.AI、cs.LG

2501.03271 2025-01-22 cs.LG cs.AI cs.CL 87%

DPO Kernels: A Semantically-Aware, Kernel-Enhanced, and Divergence-Rich Paradigm for Direct Preference Optimization

Amitava Das, Suranjana Trivedy, Danush Khanna, Rajarshi Roy, Gurpreet Singh, Basab Ghosh, Yaswanth Narsupalli, Vinija Jain, Vasu Sharma, Aishwarya Naresh Reganti, Aman Chadha

专题命中偏好对齐：DPO(title,abstract)；alignment(abstract)；safety(abstract)；分类 cs.CL、cs.AI、cs.LG

2411.14797 2024-11-25 cs.LG cs.AI cs.CL cs.CV 87%

Continual SFT Matches Multimodal RLHF with Negative Supervision

Ke Zhu, Yu Wang, Yanpeng Sun, Qiang Chen, Jiangjiang Liu, Gang Zhang, Jingdong Wang

专题命中偏好对齐：RLHF(title,abstract)；alignment(abstract)；DPO(abstract)；分类 cs.CL、cs.AI、cs.LG

2411.01493 2024-11-12 cs.LG cs.AI cs.CL 87%

Sample-Efficient Alignment for LLMs

Zichen Liu, Changyu Chen, Chao Du, Wee Sun Lee, Min Lin

专题命中偏好对齐：alignment(title,abstract)；RLHF(abstract)；DPO(abstract)；分类 cs.CL、cs.AI、cs.LG

2410.11677 2024-10-21 cs.CL cs.AI cs.LG 87%

Understanding Likelihood Over-optimisation in Direct Alignment Algorithms

Zhengyan Shi, Sander Land, Acyr Locatelli, Matthieu Geist, Max Bartolo

专题命中偏好对齐：alignment(title,abstract)；RLHF(abstract)；DPO(abstract)；分类 cs.CL、cs.AI、cs.LG

Comments Preprint Version

2410.06293 2024-10-10 cs.LG cs.AI cs.CL 87%

Accelerated Preference Optimization for Large Language Model Alignment

Jiafan He, Huizhuo Yuan, Quanquan Gu

专题命中偏好对齐：alignment(title,abstract)；RLHF(abstract)；DPO(abstract)；分类 cs.CL、cs.AI、cs.LG

Comments 44 pages, 10 tables

2405.00675 2024-10-08 cs.LG cs.AI cs.CL stat.ML 87%

Self-Play Preference Optimization for Language Model Alignment

Yue Wu, Zhiqing Sun, Huizhuo Yuan, Kaixuan Ji, Yiming Yang, Quanquan Gu

专题命中偏好对齐：alignment(title,abstract)；RLHF(abstract)；DPO(abstract)；分类 cs.CL、cs.AI、cs.LG

Comments 27 pages, 4 figures, 5 tables

2405.01481 2024-09-04 cs.CL cs.AI cs.LG 87%

NeMo-Aligner: Scalable Toolkit for Efficient Model Alignment

Gerald Shen, Zhilin Wang, Olivier Delalleau, Jiaqi Zeng, Yi Dong, Daniel Egert, Shengyang Sun, Jimmy Zhang, Sahil Jain, Ali Taghibakhshi, Markel Sanz Ausin, Ashwath Aithal, Oleksii Kuchaiev

专题命中偏好对齐：alignment(title,abstract)；RLHF(abstract)；DPO(abstract)；分类 cs.CL、cs.AI、cs.LG

Comments 16 pages, 4 figures, Accepted to COLM 2024

2406.15567 2024-06-25 cs.LG cs.AI cs.CL stat.ML 87%

SAIL: Self-Improving Efficient Online Alignment of Large Language Models

Mucong Ding, Souradip Chakraborty, Vibhu Agrawal, Zora Che, Alec Koppel, Mengdi Wang, Amrit Bedi, Furong Huang

专题命中偏好对齐：alignment(title,abstract)；RLHF(abstract)；DPO(abstract)；分类 cs.CL、cs.AI、cs.LG

Comments 24 pages, 6 figures, 3 tables

2402.15018 2024-06-10 cs.CL cs.CY cs.LG 87%

Unintended Impacts of LLM Alignment on Global Representation

Michael J. Ryan, William Held, Diyi Yang

专题命中偏好对齐：alignment(title,abstract)；RLHF(abstract)；DPO(abstract)；分类 cs.CL、cs.CY、cs.LG

Comments Accepted to ACL 2024 main conference

2405.21046 2024-06-03 cs.LG cs.AI cs.CL stat.ML 87%

Exploratory Preference Optimization: Harnessing Implicit Q*-Approximation for Sample-Efficient RLHF

Tengyang Xie, Dylan J. Foster, Akshay Krishnamurthy, Corby Rosset, Ahmed Awadallah, Alexander Rakhlin

专题命中偏好对齐：RLHF(title,abstract)；alignment(abstract)；DPO(abstract)；分类 cs.CL、cs.AI、cs.LG

2404.00604 2024-04-02 cs.CL cs.AI cs.LG 87%

Extensive Self-Contrast Enables Feedback-Free Language Model Alignment

Xiao Liu, Xixuan Song, Yuxiao Dong, Jie Tang

专题命中偏好对齐：alignment(title,abstract)；RLHF(abstract)；DPO(abstract)；分类 cs.CL、cs.AI、cs.LG

2511.09105 2025-11-13 cs.LG cs.AI 87%

Cost-Minimized Label-Flipping Poisoning Attack to LLM Alignment

Shigeki Kusaka, Keita Saito, Mikoto Kudo, Takumi Tanabe, Akifumi Wachi, Youhei Akimoto

专题命中偏好对齐：alignment(title,abstract)；RLHF(abstract)；DPO(abstract)；分类 cs.AI、cs.LG

Comments accepted for AAAI 2026 Special Track on AI Alignment

2506.06444 2025-07-10 cs.LG cs.AI cs.CR 87%

Saffron-1: Safety Inference Scaling

Ruizhong Qiu, Gaotang Li, Tianxin Wei, Jingrui He, Hanghang Tong

机构 * University of Illinois Urbana–Champaign（伊利诺伊大学厄巴纳-香槟分校）

专题命中偏好对齐：safety(title,abstract)；alignment(abstract)；jailbreak(abstract)；分类 cs.AI、cs.LG

Comments Previous title: "Saffron-1: Towards an Inference Scaling Paradigm for LLM Safety Assurance"

2402.08005 2024-02-14 cs.CL cs.LG 87%

Refined Direct Preference Optimization with Synthetic Data for Behavioral Alignment of LLMs

Víctor Gallego

专题命中偏好对齐：alignment(title,abstract)；DPO(abstract)；safety(abstract)；分类 cs.CL、cs.LG

Comments Pre-print. Submitted to the ICLR 2024 Workshop on Representational Alignment (Re-Align)

2509.20357 2026-08-18 cs.CL 版本更新 86%

Language Models that Think, Chat Better

能思考、对话更出色的语言模型

Adithya Bhaskar, Xi Ye, Danqi Chen

机构 * Princeton Language and Intelligence（普林斯顿语言与智能研究所）； Princeton University（普林斯顿大学）

专题命中偏好对齐：RLHF(summary_cn,abstract)；DPO(abstract,abstract_cn)；分类 cs.CL

AI总结本文提出RLMT方法，将RLVR扩展至开放式任务，使语言模型生成长CoT推理，在多基准测试中优于RLHF，仅用少量提示训练的基础模型性能超多阶段后训练的指令微调模型

Comments COLM 2026; we release our code, data, and artifacts publicly at https://github.com/princeton-pli/RLMT

2606.08940 2026-06-09 cs.CL 新提交 86%

Multilingual Sentiment Aware Text Summarization A Reinforcement Learning Approach for Consistency Maintenance

多语言情感感知文本摘要：一种用于一致性维护的强化学习方法

Mikhail Krasitskii, Alexander Gelbukh, Olga Kolesnikova, Grigori Sidorov

机构 * Instituto Politécnico Nacional (IPN), Centro de Investigación en Computación (CIC)（国立理工学院（IPN），计算研究中心（CIC））

专题命中偏好对齐：RLHF(summary_cn,abstract)；alignment(abstract)；safety(abstract)；分类 cs.CL

AI总结研究RLHF摘要中的情感漂移现象，提出基于策略归因框架的情感感知KL正则化方法，在保持摘要质量的同时缓解情感中性化。

2407.16216 2026-05-19 cs.CL 86%

Reinforcement Learning for LLM Post-Training: A Survey

为大型语言模型进行后训练的强化学习：综述

Zhichao Wang, Kiran Ramnath, Bin Bi, Shiva Kumar Pentyala, Sougata Chaudhuri, Shubham Mehrotra, Zixu, Zhu, Xiang-Bo Mao, Sitaram Asur, Na, Cheng

机构 * Salesforce ； AWS AI Labs ； Airbnb

专题命中偏好对齐：RLHF(summary_cn,abstract)；DPO(abstract,abstract_cn)；分类 cs.CL

AI总结本文综述了通过强化学习进行大型语言模型后训练的方法，分析了RLHF和RLVR等技术，并提出统一的策略梯度框架，旨在为研究人员提供技术参考。

2604.24395 2026-04-28 cs.AI 86%

Aligning with Your Own Voice: Self-Corrected Preference Learning for Hallucination Mitigation in LVLMs

以自身声音对齐：用于减轻大型视觉-语言模型幻觉的自我纠正偏好学习

Byeonggeuk Lim, JungMin Yun, Junehyoung Kwon, Kyeonghyun Kim, YoungBin Kim

机构 * Graduate School of Advanced Imaging Sciences, Multimedia and Film（高级影像科学研究生院，多媒体与电影系）； Department of Artificial Intelligence（人工智能系）

专题命中偏好对齐：DPO(summary_cn,abstract)；alignment(abstract,abstract_cn)；分类 cs.AI

AI总结本文提出AVES-DPO框架，通过内在知识生成分布数据，利用共识验证机制诊断幻觉并引导模型自我纠正，有效缓解LVLMs的幻觉问题，仅需5200样本即优于现有基线。

Comments Accepted to ACL 2026

1. 偏好对齐 3252 篇

SPINAL -- Scaling-law and Preference Integration in Neural Alignment Layers

Mix- and MoE-DPO: A Variational Inference Approach to Direct Preference Optimization

TreeBoN: Enhancing Inference-Time Alignment with Speculative Tree-Search and Best-of-N Sampling

AlignDistil: Token-Level Language Model Alignment as Adaptive Policy Distillation

RLHS: Mitigating Misalignment in RLHF with Hindsight Simulation

RACE-Align: Retrieval-Augmented and Chain-of-Thought Enhanced Preference Alignment for Large Language Models

Multi-Step Alignment as Markov Games: An Optimistic Online Gradient Descent Approach with Convergence Guarantees

Moral Alignment for LLM Agents

A Common Pitfall of Margin-based Language Model Alignment: Gradient Entanglement

Bi-Factorial Preference Optimization: Balancing Safety-Helpfulness in Language Models

Enhancing Safety in Reinforcement Learning with Human Feedback via Rectified Policy Optimization

Correcting the Mythos of KL-Regularization: Direct Alignment without Overoptimization via Chi-Squared Preference Optimization

DPO Kernels: A Semantically-Aware, Kernel-Enhanced, and Divergence-Rich Paradigm for Direct Preference Optimization

Continual SFT Matches Multimodal RLHF with Negative Supervision

Sample-Efficient Alignment for LLMs

Understanding Likelihood Over-optimisation in Direct Alignment Algorithms

Accelerated Preference Optimization for Large Language Model Alignment

Self-Play Preference Optimization for Language Model Alignment

NeMo-Aligner: Scalable Toolkit for Efficient Model Alignment

SAIL: Self-Improving Efficient Online Alignment of Large Language Models

Unintended Impacts of LLM Alignment on Global Representation

Exploratory Preference Optimization: Harnessing Implicit Q*-Approximation for Sample-Efficient RLHF

Extensive Self-Contrast Enables Feedback-Free Language Model Alignment

Cost-Minimized Label-Flipping Poisoning Attack to LLM Alignment

Saffron-1: Safety Inference Scaling

Refined Direct Preference Optimization with Synthetic Data for Behavioral Alignment of LLMs

Language Models that Think, Chat Better

Multilingual Sentiment Aware Text Summarization A Reinforcement Learning Approach for Consistency Maintenance

Reinforcement Learning for LLM Post-Training: A Survey

Aligning with Your Own Voice: Self-Corrected Preference Learning for Hallucination Mitigation in LVLMs