大模型对齐与安全 - arXivDaily 专题

2511.01083 2025-11-04 cs.RO 82%

Deployable Vision-driven UAV River Navigation via Human-in-the-loop Preference Alignment

Zihan Wang, Jianwen Li, Li-Fan Wu, Nina Mahmoudian

机构 * School of Mechanical Engineering, Purdue University（机械工程学院，普渡大学）

专题命中偏好对齐：alignment(title,abstract)；safety(abstract)

Comments Submitted to ICRA 2026

2510.27254 2025-11-03 cs.CL cs.AI cs.LG 82%

Languages are Modalities: Cross-Lingual Alignment via Encoder Injection

Rajan Agarwal, Aarush Gupta

机构 * University of Waterloo（多伦多大学）

专题命中偏好对齐：alignment(title,abstract)；分类 cs.CL、cs.AI、cs.LG

Comments 14 pages, 3 Figures

2505.05465 2025-10-28 cs.CL cs.AI cs.LG 82%

ComPO: Preference Alignment via Comparison Oracles

Peter Chen, Xi Chen, Wotao Yin, Tianyi Lin

机构 * Columbia University（哥伦比亚大学）； Stern School of Business（斯特恩商学院）； New York University（纽约大学）； DAMO Academy, Alibaba Group US（阿里云达摩院）

专题命中偏好对齐：alignment(title,abstract)；分类 cs.CL、cs.AI、cs.LG

Comments Accepted to NeurIPS 2025

2505.18126 2025-09-30 cs.LG cs.AI cs.CL 82%

Reward Model Overoptimisation in Iterated RLHF

Lorenz Wolf, Robert Kirk, Mirco Musolesi

机构 * UCL Centre for Artificial Intelligence（伦敦大学学院人工智能中心）； Department of Computer Science（计算机科学系）； University College London（伦敦大学学院）； UK AI Security Institute（英国人工智能安全研究所）； Department of Computer Science and Engineering（计算机科学与工程系）； University of Bologna（博洛尼亚大学）

专题命中偏好对齐：RLHF(title,abstract)；分类 cs.CL、cs.AI、cs.LG

Comments 24 pages, 17 figures, 6 tables

2504.12083 2025-09-30 cs.CV 82%

Self-alignment of Large Video Language Models with Refined Regularized Preference Optimization

Pritam Sarkar, Ali Etemad

机构 * Queen’s University（女王大学）； Vector Institute（向量研究所）

专题命中偏好对齐：alignment(title,abstract)；DPO(abstract)

Comments NeurIPS 2025

2509.20938 2025-09-26 cs.RO cs.CV 82%

Autoregressive End-to-End Planning with Time-Invariant Spatial Alignment and Multi-Objective Policy Refinement

Jianbo Zhao, Taiyu Ban, Xiangjie Li, Xingtai Gui, Hangning Zhou, Lei Liu, Hongwei Zhao, Bin Li

机构 * University of Science and Technology of China（中国科学技术大学）； Mach Drive（马车驱动）

专题命中偏好对齐：alignment(title,abstract)；DPO(abstract)

2508.05165 2025-08-08 cs.LG cs.AI cs.CL 82%

Aligning LLMs on a Budget: Inference-Time Alignment with Heuristic Reward Models

Mason Nakamura, Saaduddin Mahmud, Kyle H. Wray, Hamed Zamani, Shlomo Zilberstein

专题命中偏好对齐：alignment(title,abstract)；分类 cs.CL、cs.AI、cs.LG

2507.20133 2025-07-30 cs.CL cs.AI cs.LG 82%

Sem-DPO: Mitigating Semantic Inconsistency in Preference Optimization for Prompt Engineering

Anas Mohamed, Azal Ahmad Khan, Xinran Wang, Ahmad Faraz Khan, Shuwen Ge, Saman Bahzad Khan, Ayaan Ahmad, Ali Anwar

机构 * University of Minnesota（明尼苏达大学）； Virginia Tech（弗吉尼亚理工大学）； Xi’an University of Technology（西安理工大学）； Lahore University of Management Sciences（拉合尔管理科学大学）； University of California, Santa Cruz（加州大学圣克鲁兹分校）

专题命中偏好对齐：DPO(title,abstract)；分类 cs.CL、cs.AI、cs.LG

2507.00018 2025-07-08 cs.LG cs.AI cs.CL 82%

Implicit Reward as the Bridge: A Unified View of SFT and DPO Connections

Bo Wang, Qinyuan Cheng, Runyu Peng, Rong Bao, Peiji Li, Qipeng Guo, Linyang Li, Zhiyuan Zeng, Yunhua Zhou, Xipeng Qiu

机构 * School of Computer Science, Fudan University（复旦大学计算机学院）； Shanghai Artificial Intelligence Laboratory（上海人工智能实验室）

专题命中偏好对齐：DPO(title,abstract)；分类 cs.CL、cs.AI、cs.LG

2507.01492 2025-07-03 cs.CV 82%

AVC-DPO: Aligned Video Captioning via Direct Preference Optimization

Jiyang Tang, Hengyi Li, Yifan Du, Wayne Xin Zhao

机构 * Gaoling School of Artificial Intelligence, Renmin University of China（中国人民大学人工智能学院）； College of Artificial Intelligence, Nankai University（南开大学人工智能学院）； School of Computer Science and Technology, Beijing Institute of Technology（北京理工大学计算机科学与技术学院）

专题命中偏好对齐：DPO(title,abstract)；alignment(abstract)

2506.15717 2025-06-23 cs.LG cs.AI cs.CL 82%

daDPO: Distribution-Aware DPO for Distilling Conversational Abilities

Zhengze Zhang, Shiqi Wang, Yiqun Shen, Simin Guo, Dahua Lin, Xiaoliang Wang, Nguyen Cam-Tu, Fei Tan

机构 * State Key Laboratory for Novel Software Technology, Nanjing University（南京大学新型软件技术国家重点实验室）； School of Artificial Intelligence, Nanjing University（南京大学人工智能学院）； University of Chicago（芝加哥大学）； The Chinese University of Hong Kong（香港中文大学）； East China Normal University（华东师范大学）

专题命中偏好对齐：DPO(title,abstract)；分类 cs.CL、cs.AI、cs.LG

2505.17017 2025-06-11 cs.CV cs.AI cs.CL cs.LG 82%

Delving into RL for Image Generation with CoT: A Study on DPO vs. GRPO

Chengzhuo Tong, Ziyu Guo, Renrui Zhang, Wenyu Shan, Xinyu Wei, Zhenghao Xing, Hongsheng Li, Pheng-Ann Heng

机构 * CUHK（香港中文大学）； MiuLar Lab（MiuLar 实验室）； MMLab ； Peking University（北京大学）； Shanghai AI Lab（上海人工智能实验室）

专题命中偏好对齐：DPO(title,abstract)；分类 cs.CL、cs.AI、cs.LG

Comments Code is released at https://github.com/ZiyuGuo99/Image-Generation-CoT

2506.05260 2025-06-06 cs.CV 82%

LeanPO: Lean Preference Optimization for Likelihood Alignment in Video-LLMs

Xiaodong Wang, Jinfa Huang, Li Yuan, Peixi Peng

机构 * Peking University（北京大学）； Peng Cheng Laboratory（鹏城实验室）

专题命中偏好对齐：alignment(title,abstract)；DPO(abstract)

Comments Code: https://github.com/Wang-Xiaodong1899/LeanPO

2501.04568 2025-05-20 cs.CV cs.AI cs.CL cs.LG 82%

Feedback-Driven Vision-Language Alignment with Minimal Human Supervision

Giorgio Giannone, Ruoteng Li, Qianli Feng, Evgeny Perevodchikov, Rui Chen, Aleix Martinez

专题命中偏好对齐：alignment(title,abstract)；分类 cs.CL、cs.AI、cs.LG

Comments Preprint

2505.10597 2025-05-20 cs.LG cs.AI cs.CL 82%

Two Minds Better Than One: Collaborative Reward Modeling for LLM Alignment

Jiazheng Zhang, Wenqing Jing, Zizhuo Zhang, Zhiheng Xi, Shihan Dou, Rongxiang Weng, Jiahuan Li, Jingang Wang, Mingxu Chai, Shibo Hong, Tao Gui, Qi Zhang

机构 * NLP Group, Fudan University（复旦大学自然语言处理组）； Hong Kong Baptist University（香港 Baptist 大学）； LLM Team, Meituan Inc（美团公司大语言模型团队）

专题命中偏好对齐：alignment(title,abstract)；分类 cs.CL、cs.AI、cs.LG

2505.07271 2025-05-13 cs.CL cs.AI cs.LG 82%

On the Robustness of Reward Models for Language Model Alignment

Jiwoo Hong, Noah Lee, Eunki Kim, Guijin Son, Woojin Chung, Aman Gupta, Shao Tang, James Thorne

专题命中偏好对齐：alignment(title)；RLHF(abstract)；分类 cs.CL、cs.AI、cs.LG

Comments ICML 2025

2406.14088 2025-04-25 cs.DC cs.AI cs.CL cs.LG 82%

ReaL: Efficient RLHF Training of Large Language Models with Parameter Reallocation

Zhiyu Mei, Wei Fu, Kaiwei Li, Guangju Wang, Huanchen Zhang, Yi Wu

专题命中偏好对齐：RLHF(title,abstract)；分类 cs.CL、cs.AI、cs.LG

Comments 11 pages (20 pages with references and the appendix), 17 figures. Accepted by MLSys 25

2504.10329 2025-04-22 cs.CV 82%

InstructEngine: Instruction-driven Text-to-Image Alignment

Xingyu Lu, Yuhang Hu, YiFan Zhang, Kaiyu Jiang, Changyi Liu, Tianke Zhang, Jinpeng Wang, Chun Yuan, Bin Wen, Fan Yang, Tingting Gao, Di Zhang

机构 * Tsinghua University（清华大学）； Kuaishou Technology（快手科技）； CASIA（中国科学院自动化研究所）

专题命中偏好对齐：alignment(title,abstract)；RLHF(abstract)

Comments 8 pages, 7 figures

2407.07880 2025-04-21 cs.LG cs.AI cs.CL 82%

Towards Robust Alignment of Language Models: Distributionally Robustifying Direct Preference Optimization

Junkang Wu, Yuexiang Xie, Zhengyi Yang, Jiancan Wu, Jiawei Chen, Jinyang Gao, Bolin Ding, Xiang Wang, Xiangnan He

专题命中偏好对齐：alignment(title)；DPO(abstract)；分类 cs.CL、cs.AI、cs.LG

2503.16480 2025-03-24 cs.HC cs.AI cs.CL cs.CY 82%

Human Preferences for Constructive Interactions in Language Model Alignment

Yara Kyrychenko, Jon Roozenbeek, Brandon Davidson, Sander van der Linden, Ramit Debnath

专题命中偏好对齐：alignment(title,abstract)；分类 cs.CL、cs.AI、cs.CY

Comments 1 Figure, 1 Table, 11 pages

2503.01864 2025-03-05 cs.LG cs.AI cs.CL 82%

Larger or Smaller Reward Margins to Select Preferences for Alignment?

Kexin Huang, Junkang Wu, Ziqian Chen, Xue Wang, Jinyang Gao, Bolin Ding, Jiancan Wu, Xiangnan He, Xiang Wang

专题命中偏好对齐：alignment(title,abstract)；分类 cs.CL、cs.AI、cs.LG

2407.06057 2025-03-05 cs.CL cs.AI cs.LG 82%

Variational Best-of-N Alignment

Afra Amini, Tim Vieira, Elliott Ash, Ryan Cotterell

专题命中偏好对齐：alignment(title,abstract)；分类 cs.CL、cs.AI、cs.LG

Comments Accepted at ICLR 2025

2406.04412 2025-03-05 cs.LG cs.AI cs.CL 82%

Spread Preference Annotation: Direct Preference Judgment for Efficient LLM Alignment

Dongyoung Kim, Kimin Lee, Jinwoo Shin, Jaehyung Kim

专题命中偏好对齐：alignment(title,abstract)；分类 cs.CL、cs.AI、cs.LG

Comments ICLR 2025 Oral Presentation, 22 pages

2501.09695 2025-03-04 cs.CV 82%

Mitigating Hallucinations in Large Vision-Language Models via DPO: On-Policy Data Hold the Key

Zhihe Yang, Xufang Luo, Dongqi Han, Yunjian Xu, Dongsheng Li

专题命中偏好对齐：DPO(title,abstract)；alignment(abstract)

Comments Accepted by CVPR 2025

2502.18411 2025-03-04 cs.CV 82%

OmniAlign-V: Towards Enhanced Alignment of MLLMs with Human Preference

Xiangyu Zhao, Shengyuan Ding, Zicheng Zhang, Haian Huang, Maosong Cao, Weiyun Wang, Jiaqi Wang, Xinyu Fang, Wenhai Wang, Guangtao Zhai, Haodong Duan, Hua Yang, Kai Chen

专题命中偏好对齐：alignment(title,abstract)；DPO(abstract)

2502.20356 2025-02-28 cs.CL cs.AI cs.LG 82%

Bridging the Creativity Understanding Gap: Small-Scale Human Alignment Enables Expert-Level Humor Ranking in LLMs

Kuan Lok Zhou, Jiayi Chen, Siddharth Suresh, Reuben Narad, Timothy T. Rogers, Lalit K Jain, Robert D Nowak, Bob Mankoff, Jifan Zhang

专题命中偏好对齐：alignment(title,abstract)；分类 cs.CL、cs.AI、cs.LG

2501.00560 2025-02-12 cs.CL cs.AI cs.LG 82%

Re-evaluating Automatic LLM System Ranking for Alignment with Human Preference

Mingqi Gao, Yixin Liu, Xinyu Hu, Xiaojun Wan, Jonathan Bragg, Arman Cohan

专题命中偏好对齐：alignment(title,abstract)；分类 cs.CL、cs.AI、cs.LG

Comments Findings of NAACL 2025

2405.15973 2025-02-11 cs.CV cs.AI cs.CL cs.LG 82%

Enhancing Visual-Language Modality Alignment in Large Vision Language Models via Self-Improvement

Xiyao Wang, Jiuhai Chen, Zhaoyang Wang, Yuhang Zhou, Yiyang Zhou, Huaxiu Yao, Tianyi Zhou, Tom Goldstein, Parminder Bhatia, Furong Huang, Cao Xiao

专题命中偏好对齐：alignment(title,abstract)；分类 cs.CL、cs.AI、cs.LG

Comments NAACL 2025 Findings

2403.09472 2024-12-11 cs.LG cs.AI cs.CL 82%

Easy-to-Hard Generalization: Scalable Alignment Beyond Human Supervision

Zhiqing Sun, Longhui Yu, Yikang Shen, Weiyang Liu, Yiming Yang, Sean Welleck, Chuang Gan

专题命中偏好对齐：alignment(title,abstract)；分类 cs.CL、cs.AI、cs.LG

Comments Accepted at NeurIPS 2024

2405.07863 2024-11-13 cs.LG cs.AI cs.CL stat.ML 82%

RLHF Workflow: From Reward Modeling to Online RLHF

Hanze Dong, Wei Xiong, Bo Pang, Haoxiang Wang, Han Zhao, Yingbo Zhou, Nan Jiang, Doyen Sahoo, Caiming Xiong, Tong Zhang

专题命中偏好对齐：RLHF(title,abstract)；分类 cs.CL、cs.AI、cs.LG

Comments Published in Transactions on Machine Learning Research (09/2024)

1. 偏好对齐 3261 篇

Deployable Vision-driven UAV River Navigation via Human-in-the-loop Preference Alignment

Languages are Modalities: Cross-Lingual Alignment via Encoder Injection

ComPO: Preference Alignment via Comparison Oracles

Reward Model Overoptimisation in Iterated RLHF

Self-alignment of Large Video Language Models with Refined Regularized Preference Optimization

Autoregressive End-to-End Planning with Time-Invariant Spatial Alignment and Multi-Objective Policy Refinement

Aligning LLMs on a Budget: Inference-Time Alignment with Heuristic Reward Models

Sem-DPO: Mitigating Semantic Inconsistency in Preference Optimization for Prompt Engineering

Implicit Reward as the Bridge: A Unified View of SFT and DPO Connections

AVC-DPO: Aligned Video Captioning via Direct Preference Optimization

daDPO: Distribution-Aware DPO for Distilling Conversational Abilities

Delving into RL for Image Generation with CoT: A Study on DPO vs. GRPO

LeanPO: Lean Preference Optimization for Likelihood Alignment in Video-LLMs

Feedback-Driven Vision-Language Alignment with Minimal Human Supervision

Two Minds Better Than One: Collaborative Reward Modeling for LLM Alignment

On the Robustness of Reward Models for Language Model Alignment

ReaL: Efficient RLHF Training of Large Language Models with Parameter Reallocation

InstructEngine: Instruction-driven Text-to-Image Alignment

Towards Robust Alignment of Language Models: Distributionally Robustifying Direct Preference Optimization

Human Preferences for Constructive Interactions in Language Model Alignment

Larger or Smaller Reward Margins to Select Preferences for Alignment?

Variational Best-of-N Alignment

Spread Preference Annotation: Direct Preference Judgment for Efficient LLM Alignment

Mitigating Hallucinations in Large Vision-Language Models via DPO: On-Policy Data Hold the Key

OmniAlign-V: Towards Enhanced Alignment of MLLMs with Human Preference

Bridging the Creativity Understanding Gap: Small-Scale Human Alignment Enables Expert-Level Humor Ranking in LLMs

Re-evaluating Automatic LLM System Ranking for Alignment with Human Preference

Enhancing Visual-Language Modality Alignment in Large Vision Language Models via Self-Improvement

Easy-to-Hard Generalization: Scalable Alignment Beyond Human Supervision

RLHF Workflow: From Reward Modeling to Online RLHF