大模型对齐与安全 - arXivDaily 专题

2508.15044 2025-09-24 cs.CL 77%

Reward-Shifted Speculative Sampling Is An Efficient Test-Time Weak-to-Strong Aligner

Bolian Li, Yanran Wu, Xinyu Luo, Ruqi Zhang

机构 * Department of Computer Science, Purdue University（计算机科学系，普渡大学）

专题命中偏好对齐：alignment(abstract)；RLHF(abstract)；safety(abstract)；分类 cs.CL

Comments EMNLP 2025 Main Conference

2502.13146 2025-09-23 cs.CV cs.LG 77%

Re-Align: Aligning Vision Language Models via Retrieval-Augmented Direct Preference Optimization

Shuo Xing, Peiran Li, Yuping Wang, Ruizheng Bai, Yueqi Wang, Chan-Wei Hu, Chengxuan Qian, Huaxiu Yao, Zhengzhong Tu

机构 * Texas A&M University（德克萨斯大学）； University of Michigan（密歇根大学）； UIUC（伊利诺伊大学香槟分校）； UNC Chapel Hill（北卡罗来纳大学教堂山分校）

专题命中偏好对齐：alignment(abstract)；RLHF(abstract)；DPO(abstract)；分类 cs.LG

Comments Published at EMNLP 2025

2509.03730 2025-09-08 cs.AI cs.CL cs.CY cs.LG stat.ML 77%

The Personality Illusion: Revealing Dissociation Between Self-Reports & Behavior in LLMs

Pengrui Han, Rafal Kocielnik, Peiyang Song, Ramit Debnath, Dean Mobbs, Anima Anandkumar, R. Michael Alvarez

机构 * Caltech（加州理工学院）； UIUC（伊利诺伊大学）； University of Cambridge（剑桥大学）

专题命中偏好对齐：alignment(abstract)；RLHF(abstract)；分类 cs.CL、cs.AI、cs.CY

Comments We make public all code and source data at https://github.com/psychology-of-AI/Personality-Illusion for full reproducibility

2508.12458 2025-08-19 cs.CL 77%

M3PO: Multimodal-Model-Guided Preference Optimization for Visual Instruction Following

Ruirui Gao, Emily Johnson, Bowen Tan, Yanfei Qian

机构 * University of Massachusetts, Amherst（马萨诸塞大学阿姆赫斯特分校）

专题命中偏好对齐：alignment(abstract)；RLHF(abstract)；DPO(abstract)；分类 cs.CL

2506.14336 2025-06-18 cs.AI 77%

AviationLLM: An LLM-based Knowledge System for Aviation Training

Jia'ang Wan, Feng Shen, Fujuan Li, Yanjin Sun, Yan Li, Shiwen Zhang

机构 * Wuhan University（武汉大学）； China Eastern Technology Application Research And Development Center Co., Ltd（中国东方技术应用研发有限公司）

专题命中偏好对齐：alignment(abstract)；DPO(abstract)；safety(abstract)；分类 cs.AI

2506.07035 2025-06-10 q-bio.BM cs.AI 77%

AnnoDPO: Protein Functional Annotation Learning with Direct Preference Optimization

Zixuan Jiang, Renjing Xu

专题命中偏好对齐：alignment(abstract)；RLHF(abstract)；DPO(abstract)；分类 cs.AI

2506.01937 2025-06-03 cs.CL 77%

RewardBench 2: Advancing Reward Model Evaluation

Saumya Malik, Valentina Pyatkin, Sander Land, Jacob Morrison, Noah A. Smith, Hannaneh Hajishirzi, Nathan Lambert

机构 * Allen Institute for Artificial Intelligence（艾伦人工智能研究所）； University of Washington（华盛顿大学）； Cohere

专题命中偏好对齐：alignment(abstract)；RLHF(abstract)；safety(abstract)；分类 cs.CL

Comments Data, models, and leaderboard available at https://huggingface.co/collections/allenai/reward-bench-2-683d2612a4b3e38a3e53bb51

2505.17122 2025-05-26 cs.CL 77%

Shallow Preference Signals: Large Language Model Aligns Even Better with Truncated Data?

Xuan Qi, Jiahao Qiu, Xinzhe Juan, Yue Wu, Mengdi Wang

机构 * IIIS, Tsinghua University（清华大学信息学院）； AI Lab, Princeton University（普林斯顿大学人工智能实验室）； Department of Computer Science & Engineering, University of Michigan（密歇根大学计算机科学与工程系）

专题命中偏好对齐：alignment(abstract)；RLHF(abstract)；DPO(abstract)；分类 cs.CL

Comments 17 pages, 7 figures

2406.00888 2025-04-22 cs.CL cs.HC 77%

Aligning Language Models with Demonstrated Feedback

Omar Shaikh, Michelle S. Lam, Joey Hejna, Yijia Shao, Hyundong Cho, Michael S. Bernstein, Diyi Yang

机构 * Stanford University（斯坦福大学）； USC（美国南加州大学）

专题命中偏好对齐：alignment(abstract)；RLHF(abstract)；DPO(abstract)；分类 cs.CL

Comments ICLR 2025; 28 pages, 8 figures

2411.08923 2025-03-27 cs.CV cs.LG 77%

Aligning Visual Contrastive learning models via Preference Optimization

Amirabbas Afzali, Borna Khodabandeh, Ali Rasekh, Mahyar JafariNodeh, Sepehr kazemi, Simon Gottschalk

专题命中偏好对齐：alignment(abstract)；RLHF(abstract)；DPO(abstract)；分类 cs.LG

2503.17662 2025-03-26 cs.CL 77%

Enhancing Persona Consistency for LLMs' Role-Playing using Persona-Aware Contrastive Learning

Ke Ji, Yixin Lian, Linxu Li, Jingsheng Gao, Weiyuan Li, Bin Dai

专题命中偏好对齐：alignment(abstract)；RLHF(abstract)；safety(abstract)；分类 cs.CL

Comments 18 pages, 4 figures

2502.16182 2025-03-21 cs.CL 77%

IPO: Your Language Model is Secretly a Preference Classifier

Shivank Garg, Ayush Singh, Shweta Singh, Paras Chopra

专题命中偏好对齐：alignment(abstract)；RLHF(abstract)；DPO(abstract)；分类 cs.CL

2503.11701 2025-03-18 cs.LG 77%

A Survey of Direct Preference Optimization

Shunyu Liu, Wenkai Fang, Zetian Hu, Junjie Zhang, Yang Zhou, Kongcheng Zhang, Rongcheng Tu, Ting-En Lin, Fei Huang, Mingli Song, Yongbin Li, Dacheng Tao

专题命中偏好对齐：alignment(abstract)；RLHF(abstract)；DPO(abstract)；分类 cs.LG

2502.14340 2025-02-21 cs.CL 77%

Earlier Tokens Contribute More: Learning Direct Preference Optimization From Temporal Decay Perspective

Ruichen Shao, Bei Li, Gangao Liu, Yang Chen, Xiang Zhou, Jingang Wang, Xunliang Cai, Peng Li

专题命中偏好对齐：alignment(abstract)；RLHF(abstract)；DPO(abstract)；分类 cs.CL

Comments Accepted by ICLR 2025

2403.19443 2025-01-23 cs.CL 77%

Mixed Preference Optimization: Reinforcement Learning with Data Selection and Better Reference Model

Qi Gou, Cam-Tu Nguyen

专题命中偏好对齐：alignment(abstract)；RLHF(abstract)；DPO(abstract)；分类 cs.CL

2405.12750 2025-01-20 cs.CR cs.AI 77%

Generative AI in Cybersecurity: A Comprehensive Review of LLM Applications and Vulnerabilities

Mohamed Amine Ferrag, Fatima Alwahedi, Ammar Battah, Bilel Cherif, Abdechakour Mechri, Norbert Tihanyi, Tamas Bisztray, Merouane Debbah

专题命中偏好对齐：RLHF(abstract)；DPO(abstract)；prompt injection(abstract)；分类 cs.AI

Comments 52 pages, 8 figures

2501.06655 2025-01-14 cs.LG cs.CV 77%

Personalized Preference Fine-tuning of Diffusion Models

Meihua Dang, Anikait Singh, Linqi Zhou, Stefano Ermon, Jiaming Song

专题命中偏好对齐：alignment(abstract)；RLHF(abstract)；DPO(abstract)；分类 cs.LG

2410.23569 2025-01-10 cs.LG 77%

RA-PbRL: Provably Efficient Risk-Aware Preference-Based Reinforcement Learning

Yujie Zhao, Jose Efraim Aguilar Escamill, Weyl Lu, Huazheng Wang

专题命中偏好对齐：RLHF(abstract)；safety(abstract)；AI safety(abstract)；分类 cs.LG

2406.10957 2024-12-10 cs.CL 77%

Eliminating Biased Length Reliance of Direct Preference Optimization via Down-Sampled KL Divergence

Junru Lu, Jiazheng Li, Siyu An, Meng Zhao, Yulan He, Di Yin, Xing Sun

专题命中偏好对齐：alignment(abstract)；RLHF(abstract)；DPO(abstract)；分类 cs.CL

Comments EMNLP 2024 Main, Final Version

2410.16586 2024-10-23 cs.AI cs.IR 77%

Optimizing LLMs with Direct Preferences: A Data Efficiency Perspective

Pietro Bernardelle, Gianluca Demartini

专题命中偏好对齐：alignment(abstract)；RLHF(abstract)；DPO(abstract)；分类 cs.AI

2410.15096 2024-10-22 cs.AI 77%

GDPO: Learning to Directly Align Language Models with Diversity Using GFlowNets

Oh Joon Kwon, Daiki E. Matsunaga, Kee-Eung Kim

专题命中偏好对齐：alignment(abstract)；RLHF(abstract)；DPO(abstract)；分类 cs.AI

Journal ref EMNLP 2024

2410.14184 2024-10-21 cs.CL 77%

MetaAlign: Align Large Language Models with Diverse Preferences during Inference Time

Mozhi Zhang, Pengyu Wang, Chenkun Tan, Mianqiu Huang, Dong Zhang, Yaqian Zhou, Xipeng Qiu

专题命中偏好对齐：alignment(abstract)；RLHF(abstract)；DPO(abstract)；分类 cs.CL

Comments 19 pages, 6 figures

2404.12358 2024-08-14 cs.LG 77%

From $r$ to $Q^*$: Your Language Model is Secretly a Q-Function

Rafael Rafailov, Joey Hejna, Ryan Park, Chelsea Finn

专题命中偏好对齐：alignment(abstract)；RLHF(abstract)；DPO(abstract)；分类 cs.LG

Comments COLM 2024

2305.14718 2024-04-23 cs.CL 77%

Leftover Lunch: Advantage-based Offline Reinforcement Learning for Language Models

Ashutosh Baheti, Ximing Lu, Faeze Brahman, Ronan Le Bras, Maarten Sap, Mark Riedl

专题命中偏好对齐：alignment(abstract)；RLHF(abstract)；DPO(abstract)；分类 cs.CL

Comments published at ICLR 2024

2404.04291 2024-04-09 cs.LG 77%

Investigating Regularization of Self-Play Language Models

Reda Alami, Abdalgader Abubaker, Mastane Achab, Mohamed El Amine Seddik, Salem Lahlou

专题命中偏好对齐：alignment(abstract)；RLHF(abstract)；DPO(abstract)；分类 cs.LG

2402.14740 2024-02-27 cs.LG 77%

Back to Basics: Revisiting REINFORCE Style Optimization for Learning from Human Feedback in LLMs

Arash Ahmadian, Chris Cremer, Matthias Gallé, Marzieh Fadaee, Julia Kreutzer, Olivier Pietquin, Ahmet Üstün, Sara Hooker

专题命中偏好对齐：alignment(abstract)；RLHF(abstract)；DPO(abstract)；分类 cs.LG

Comments 27 pages, 7 figures, 2 tables

2309.06657 2024-01-25 cs.CL 77%

Statistical Rejection Sampling Improves Preference Optimization

Tianqi Liu, Yao Zhao, Rishabh Joshi, Misha Khalman, Mohammad Saleh, Peter J. Liu, Jialu Liu

专题命中偏好对齐：alignment(abstract)；RLHF(abstract)；DPO(abstract)；分类 cs.CL

Comments Accepted in ICLR 2024

2605.18840 2026-05-26 cs.LG cs.AI cs.CL 76%

The Growing Pains of Frontier Models: When Leaderboards Stop Separating and What to Measure Next

前沿模型的成长之痛：当排行榜不再区分以及接下来衡量什么

Adil Amin

机构 * Zehen Labs（泽亨实验室）

专题命中偏好对齐：RLHF(abstract,abstract_cn)；分类 cs.CL、cs.AI、cs.LG；alignment(comments)

AI总结本文通过分解SWE-bench和GPQA Diamond分数为种群耦合趋势和每版本残差（h场），诊断前沿模型能力之间的协作与权衡，并提供三步诊断法、每实验室测量优先级表及七个可证伪预测。

Comments 13 pages, 5 figures, 4 tables. Companion paper: "Lying Is Just a Phase: The Hidden Alignment Transition in Language Model Scaling." ( https://doi.org/10.48550/arXiv.2605.18838 ). Code: https://github.com/adilamin89/cape-scaling . Dashboard: https://zehenlabs.com/cape/

2505.23761 2025-10-03 cs.LG cs.AI cs.CL 76%

Differential Information Distribution: A Bayesian Perspective on Direct Preference Optimization

Yunjae Won, Hyunji Lee, Hyeonbin Hwang, Minjoon Seo

机构 * KAIST AI（韩国科学技术院人工智能研究所）

专题命中偏好对齐：DPO(abstract,comments)；alignment(abstract)；分类 cs.CL、cs.AI、cs.LG

Comments Preprint, under review. 39 pages, 12 figures. Updates from v1: Added new theoretical results on DPO training dynamics and policy exploration, included experiments with Qwen3-4B, and refined the discussion of log-margin dynamics

2411.07595 2025-06-16 cs.LG cs.AI cs.CL 76%

Entropy Controllable Direct Preference Optimization

Motoki Omura, Yasuhiro Fujita, Toshiki Kataoka

机构 * The University of Tokyo（东京大学）

专题命中偏好对齐：RLHF(abstract)；DPO(abstract)；分类 cs.CL、cs.AI、cs.LG；alignment(comments)

Comments ICML 2025 Workshop on Models of Human Feedback for AI Alignment

1. 偏好对齐 3266 篇

Reward-Shifted Speculative Sampling Is An Efficient Test-Time Weak-to-Strong Aligner

Re-Align: Aligning Vision Language Models via Retrieval-Augmented Direct Preference Optimization

The Personality Illusion: Revealing Dissociation Between Self-Reports & Behavior in LLMs

M3PO: Multimodal-Model-Guided Preference Optimization for Visual Instruction Following

AviationLLM: An LLM-based Knowledge System for Aviation Training

AnnoDPO: Protein Functional Annotation Learning with Direct Preference Optimization

RewardBench 2: Advancing Reward Model Evaluation

Shallow Preference Signals: Large Language Model Aligns Even Better with Truncated Data?

Aligning Language Models with Demonstrated Feedback

Aligning Visual Contrastive learning models via Preference Optimization

Enhancing Persona Consistency for LLMs' Role-Playing using Persona-Aware Contrastive Learning

IPO: Your Language Model is Secretly a Preference Classifier

A Survey of Direct Preference Optimization

Earlier Tokens Contribute More: Learning Direct Preference Optimization From Temporal Decay Perspective

Mixed Preference Optimization: Reinforcement Learning with Data Selection and Better Reference Model

Generative AI in Cybersecurity: A Comprehensive Review of LLM Applications and Vulnerabilities

Personalized Preference Fine-tuning of Diffusion Models

RA-PbRL: Provably Efficient Risk-Aware Preference-Based Reinforcement Learning

Eliminating Biased Length Reliance of Direct Preference Optimization via Down-Sampled KL Divergence

Optimizing LLMs with Direct Preferences: A Data Efficiency Perspective

GDPO: Learning to Directly Align Language Models with Diversity Using GFlowNets

MetaAlign: Align Large Language Models with Diverse Preferences during Inference Time

From $r$ to $Q^*$: Your Language Model is Secretly a Q-Function

Leftover Lunch: Advantage-based Offline Reinforcement Learning for Language Models

Investigating Regularization of Self-Play Language Models

Back to Basics: Revisiting REINFORCE Style Optimization for Learning from Human Feedback in LLMs

Statistical Rejection Sampling Improves Preference Optimization

The Growing Pains of Frontier Models: When Leaderboards Stop Separating and What to Measure Next

Differential Information Distribution: A Bayesian Perspective on Direct Preference Optimization

Entropy Controllable Direct Preference Optimization