大模型对齐与安全 - arXivDaily 专题

2509.10105 2025-09-17 cs.CV cs.CL 70%

VARCO-VISION-2.0 Technical Report

Young-rok Cha, Jeongho Ju, SunYoung Park, Jong-Hyeon Lee, Younghyun Yu, Youngjune Kim

机构 * NC AI

专题命中偏好对齐：alignment(abstract)；safety(abstract)；分类 cs.CL

Comments 19 pages, 1 figure, 14 tables. Technical report for VARCO-VISION-2.0, a Korean-English bilingual VLM in 14B and 1.7B variants. Key features: multi-image understanding, OCR with text localization, improved Korean capabilities

2509.00373 2025-09-03 cs.CV cs.AI 70%

Activation Steering Meets Preference Optimization: Defense Against Jailbreaks in Vision Language Models

Sihao Wu, Gaojie Jin, Wei Huang, Jianhong Wang, Xiaowei Huang

机构 * University of Liverpool（利物浦大学）； University of Exeter（埃克塞特大学）； Purple Mountain Laboratories（紫金山实验室）； University of Bristol（布里斯托大学）

专题命中偏好对齐：alignment(abstract)；safety(abstract)；分类 cs.AI

2508.19922 2025-08-28 cs.CL 70%

HEAL: A Hypothesis-Based Preference-Aware Analysis Framework

Yifu Huo, Chenglong Wang, Qiren Zhu, Shunjie Xing, Tong Xiao, Chunliang Zhang, Tongran Liu, Jinbo Zhu

机构 * School of Computer Science and Engineering, Northeastern University（计算机科学与工程学院，东北大学）； NiuTrans Research（NiuTrans研究院）； CAS Key Laboratory of Behavioral Science, Institute of Psychology, CAS（行为科学重点实验室，心理学研究所）

专题命中偏好对齐：alignment(abstract)；DPO(abstract)；分类 cs.CL

Comments Accepted by EMNLP 2025 Findings

2508.05496 2025-08-13 cs.AI 70%

InfiAlign: A Scalable and Sample-Efficient Framework for Aligning LLMs to Enhance Reasoning Capabilities

Shuo Cai, Su Lu, Qi Zhou, Kejing Yang, Zhijie Sang, Congkai Xie, Hongxia Yang

专题命中偏好对齐：alignment(abstract)；DPO(abstract)；分类 cs.AI

2506.23998 2025-08-12 cs.CL 70%

Auto-TA: Towards Scalable Automated Thematic Analysis (TA) via Multi-Agent Large Language Models with Reinforcement Learning

Seungjun Yi, Joakim Nguyen, Huimin Xu, Terence Lim, Andrew Well, Mia Markey, Ying Ding

专题命中偏好对齐：alignment(abstract)；RLHF(abstract)；分类 cs.CL

Comments Presented at ACL 2025 SRW

2508.05011 2025-08-08 cs.SD cs.AI eess.AS 70%

Towards Hallucination-Free Music: A Reinforcement Learning Preference Optimization Framework for Reliable Song Generation

Huaicheng Zhang, Wei Tan, Guangzheng Li, Yixuan Zhang, Hangting Chen, Shun Lei, Chenyu Yang, Zhiyong Wu, Shuai Wang, Qijun Huang, Dong Yu

机构 * Tencent AI Lab（腾讯AI实验室）

专题命中偏好对齐：alignment(abstract)；DPO(abstract)；分类 cs.AI

2508.02312 2025-08-05 cs.CR cs.AI 70%

A Survey on Data Security in Large Language Models

Kang Chen, Xiuze Zhou, Yuanguo Lin, Jinhe Su, Yuanhui Yu, Li Shen, Fan Lin

机构 * School of Computer Engineering, Jimei University, Xiamen, 361021, China（厦门大学计算机工程学院）； College of Science, Mathematics and Technology, Wenzhou-Kean University, Wenzhou, 325060, China（温州-凯恩大学科学、数学与技术学院）； The Hong Kong University of Science and Technology (Guangzhou), Guangzhou, 511453, China（香港科学与技术大学（广州））； School of Professional Studies, New York University, New York, 10003, United States（纽约大学专业研究学院）； School of Informatics, Xiamen University, Xiamen, 361102, China（厦门大学信息学院）

专题命中偏好对齐：RLHF(abstract)；prompt injection(abstract)；分类 cs.AI

2507.11155 2025-07-16 cs.CR cs.AI 70%

Bridging the Gap in Vision Language Models in Identifying Unsafe Concepts Across Modalities

Yiting Qu, Michael Backes, Yang Zhang

机构 * CISPA Helmholtz Center for Information Security（CISPA海德堡信息安全研究中心）

专题命中偏好对齐：alignment(abstract)；DPO(abstract)；分类 cs.AI

Comments To Appear in the 34th USENIX Security Symposium, August 2025

2503.08942 2025-07-10 cs.LG 70%

Extragradient Preference Optimization (EGPO): Beyond Last-Iterate Convergence for Nash Learning from Human Feedback

Runlong Zhou, Maryam Fazel, Simon S. Du

机构 * University of Washington（华盛顿大学）

专题命中偏好对齐：alignment(abstract)；RLHF(abstract)；分类 cs.LG

Comments COLM 2025

2502.16825 2025-07-01 cs.CL 70%

Finding the Sweet Spot: Preference Data Construction for Scaling Preference Optimization

Yao Xiao, Hai Ye, Linyao Chen, Hwee Tou Ng, Lidong Bing, Xiaoli Li, Roy Ka-wei Lee

机构 * Singapore University of Technology and Design（新加坡科技设计大学）； National University of Singapore（国立新加坡大学）； The University of Tokyo（东京大学）； MiroMind（MiroMind公司）； Institute for Infocomm Research, A*Star, Singapore（新加坡*星研究院信息与通信研究所）

专题命中偏好对齐：alignment(abstract)；DPO(abstract)；分类 cs.CL

Comments ACL25 Main

2501.17726 2025-06-26 cs.CV cs.CL 70%

VICCA: Visual Interpretation and Comprehension of Chest X-ray Anomalies in Generated Report Without Human Feedback

Sayeh Gholipour Picha, Dawood Al Chanti, Alice Caplier

机构 * Univ. Grenoble Alpes（格勒诺布尔阿尔卑斯大学）； CNRS（法国国家科学研究中心）； Grenoble INP（格勒诺布尔研究所）； GIPSA-lab（GIPSA实验室）

专题命中偏好对齐：alignment(abstract)；trustworthy(abstract)；分类 cs.CL

Journal ref Machine Learning with Applications, Volume 21, 2025, 100684, ISSN 2666-8270

2505.02686 2025-06-13 cs.CL 70%

Sailing by the Stars: A Survey on Reward Models and Learning Strategies for Learning from Rewards

Xiaobao Wu

机构 * Nanyang Technological University（南洋理工大学）

专题命中偏好对齐：RLHF(abstract)；DPO(abstract)；分类 cs.CL

Comments 36 Pages

2506.07492 2025-06-10 cs.LG stat.ML 70%

Explicit Preference Optimization: No Need for an Implicit Reward Model

Xiangkun Hu, Lemin Kong, Tong He, David Wipf

机构 * The Chinese University of Hong Kong（香港中文大学）； Amazon Web Services（亚马逊网络服务）

专题命中偏好对齐：RLHF(abstract)；DPO(abstract)；分类 cs.LG

Comments arXiv admin note: substantial text overlap with arXiv:2407.09072

2506.04463 2025-06-06 cs.CL 70%

Aligning Large Language Models with Implicit Preferences from User-Generated Content

Zhaoxuan Tan, Zheng Li, Tianyi Liu, Haodong Wang, Hyokun Yun, Ming Zeng, Pei Chen, Zhihan Zhang, Yifan Gao, Ruijie Wang, Priyanka Nigam, Bing Yin, Meng Jiang

机构 * University of Notre Dame（诺特大学）； Amazon.com Inc（亚马逊公司）

专题命中偏好对齐：alignment(abstract)；DPO(abstract)；分类 cs.CL

Comments Accepted to ACL 2025 Main Conference

2506.00569 2025-06-03 cs.LG 70%

AutoMixAlign: Adaptive Data Mixing for Multi-Task Preference Optimization in LLMs

Nicholas E. Corrado, Julian Katz-Samuels, Adithya Devraj, Hyokun Yun, Chao Zhang, Yi Xu, Yi Pan, Bing Yin, Trishul Chilimbi

机构 * University of Wisconsin–Madison（威斯康星大学麦迪逊分校）； Amazon（亚马逊）； Georgia Institute of Technology（佐治亚理工学院）

专题命中偏好对齐：alignment(abstract)；DPO(abstract)；分类 cs.LG

Comments ACL 2025, Main Conference

2501.11463 2025-06-03 cs.CL 70%

Curiosity-Driven Reinforcement Learning from Human Feedback

Haoran Sun, Yekun Chai, Shuohuan Wang, Yu Sun, Hua Wu, Haifeng Wang

机构 * Baidu Inc.（百度公司）

专题命中偏好对齐：alignment(abstract)；RLHF(abstract)；分类 cs.CL

Comments ACL 2025

2410.19133 2025-06-03 cs.CL 70%

Hybrid Preferences: Learning to Route Instances for Human vs. AI Feedback

Lester James V. Miranda, Yizhong Wang, Yanai Elazar, Sachin Kumar, Valentina Pyatkin, Faeze Brahman, Noah A. Smith, Hannaneh Hajishirzi, Pradeep Dasigi

专题命中偏好对齐：alignment(abstract)；safety(abstract)；分类 cs.CL

Comments Code in https://github.com/allenai/hybrid-preferences, MultiPref dataset in https://huggingface.co/datasets/allenai/multipref, Updated related work and acknowledgments

2505.24199 2025-06-02 cs.CL 70%

Intuitionistic Fuzzy Sets for Large Language Model Data Annotation: A Novel Approach to Side-by-Side Preference Labeling

Yimin Du

专题命中偏好对齐：RLHF(abstract)；DPO(abstract)；分类 cs.CL

Comments 7 pages

2502.17057 2025-05-30 cs.IR cs.AI 70%

ExpandR: Teaching Dense Retrievers Beyond Queries with LLM Guidance

Sijia Yao, Pengcheng Huang, Zhenghao Liu, Yu Gu, Yukun Yan, Shi Yu, Ge Yu

机构 * Department of Computer Science and Technology, Northeastern University, China（东北大学计算机科学与技术系）； Department of Computer Science and Technology, Institute for AI, Tsinghua University, China（清华大学人工智能研究院计算机科学与技术系）； Beijing National Research Center for Information Science and Technology, China（北京信息科学与技术国家研究中心）

专题命中偏好对齐：alignment(abstract)；DPO(abstract)；分类 cs.AI

Comments 16 pages, 10 tables, 5 figures

2505.21061 2025-05-28 cs.CV cs.AI 70%

LPOI: Listwise Preference Optimization for Vision Language Models

Fatemeh Pesaran Zadeh, Yoojin Oh, Gunhee Kim

专题命中偏好对齐：RLHF(abstract)；DPO(abstract)；分类 cs.AI

Comments ACL 2025 Main. Code is released at https://github.com/fatemehpesaran310/lpoi

2505.19100 2025-05-27 cs.CL cs.CV 70%

ASPO: Adaptive Sentence-Level Preference Optimization for Fine-Grained Multimodal Reasoning

Yeyuan Wang, Dehong Gao, Rujiao Long, Lei Yi, Linbo Jin, Libin Yang, Xiaoyan Cai

机构 * Northwestern Polytechnical University, School of Automation, Xi’an, China（西北工业大学自动化学院，西安，中国）； Northwestern Polytechnical University, School of Cybersecurity, Xi’an, China（西北工业大学网络安全学院，西安，中国）； Binjiang Institute of Artificial Intelligence, ZJUT, Hangzhou, China（浙大宁波理工学院滨海人工智能研究院，杭州，中国）； Alibaba Group, Hangzhou, China（阿里巴巴集团，杭州，中国）

专题命中偏好对齐：alignment(abstract)；DPO(abstract)；分类 cs.CL

Comments Accepted by ACL 2025 findings

2505.14946 2025-05-22 cs.AI 70%

Reinforcement Learning from User Feedback

Eric Han, Jun Chen, Karthik Abinav Sankararaman, Xiaoliang Peng, Tengyu Xu, Eryk Helenowski, Kaiyan Peng, Mrinal Kumar, Sinong Wang, Han Fang, Arya Talebzadeh

机构 * Meta GenAI（Meta 生成式人工智能）

专题命中偏好对齐：RLHF(abstract)；safety(abstract)；分类 cs.AI

2412.20299 2025-05-14 cs.CL 70%

No Preference Left Behind: Group Distributional Preference Optimization

Binwei Yao, Zefan Cai, Yun-Shiuan Chuang, Shanglin Yang, Ming Jiang, Diyi Yang, Junjie Hu

机构 * University of Wisconsin-Madison（威斯康星大学麦迪逊分校）； Indiana University Indianapolis（印第安纳大学印第安纳波利斯分校）； Stanford University（斯坦福大学）

专题命中偏好对齐：alignment(abstract)；DPO(abstract)；分类 cs.CL

2411.03700 2025-05-13 cs.CL 70%

The Root Shapes the Fruit: On the Persistence of Gender-Exclusive Harms in Aligned Language Models

Anaelia Ovalle, Krunoslav Lehman Pavasovic, Louis Martin, Luke Zettlemoyer, Eric Michael Smith, Kai-Wei Chang, Adina Williams, Levent Sagun

机构 * FAIR, Meta（FAIR与Meta）； Meta United States（Meta美国分公司）； Meta France（Meta法国分公司）； Gen AI, Meta（Meta生成式人工智能部门）； UCLA United States（加州大学洛杉矶分校）

专题命中偏好对齐：alignment(abstract)；DPO(abstract)；分类 cs.CL

Comments Accepted to 2025 ACM FAccT

2505.04993 2025-05-09 cs.CL 70%

Latent Preference Coding: Aligning Large Language Models via Discrete Latent Codes

Zhuocheng Gong, Jian Guan, Wei Wu, Huishuai Zhang, Dongyan Zhao

机构 * Antiquus S. Hippocampus, Natalia Cerebro & Amelie P. Amygdale Department of Computer Science Cranberry-Lemon University Pittsburgh, PA 15213, USA（计算机科学系， Cranberry-Lemon 大学）； Ji Q. Ren & Yevgeny LeNet Department of Computational Neuroscience University of the Witwatersrand Joburg, South Africa（计算神经科学系，威特沃特斯兰大学）

专题命中偏好对齐：alignment(abstract)；DPO(abstract)；分类 cs.CL

2505.02363 2025-05-06 cs.CL 70%

SIMPLEMIX: Frustratingly Simple Mixing of Off- and On-policy Data in Language Model Preference Learning

Tianjian Li, Daniel Khashabi

机构 * Johns Hopkins University（约翰霍普金斯大学）

专题命中偏好对齐：alignment(abstract)；DPO(abstract)；分类 cs.CL

Comments To appear in ICML 2025

2504.16272 2025-04-24 cs.LG 70%

Learning Explainable Dense Reward Shapes via Bayesian Optimization

Ryan Koo, Ian Yang, Vipul Raheja, Mingyi Hong, Kwang-Sung Jun, Dongyeop Kang

机构 * University of Minnesota（明尼苏达大学）； Georgia Institute of Technology（佐治亚理工学院）； Grammarly ； University of Arizona（亚利桑那大学）

专题命中偏好对齐：alignment(abstract)；RLHF(abstract)；分类 cs.LG

2502.10482 2025-04-18 cs.AI 70%

A Self-Supervised Reinforcement Learning Approach for Fine-Tuning Large Language Models Using Cross-Attention Signals

Andrew Kiruluta, Andreas Lemos, Priscilla Burity

专题命中偏好对齐：alignment(abstract)；RLHF(abstract)；分类 cs.AI

2504.06141 2025-04-15 cs.LG 70%

Adversarial Training of Reward Models

Alexander Bukharin, Haifeng Qian, Shengyang Sun, Adithya Renduchintala, Soumye Singhal, Zhilin Wang, Oleksii Kuchaiev, Olivier Delalleau, Tuo Zhao

专题命中偏好对齐：alignment(abstract)；RLHF(abstract)；分类 cs.LG

Comments 16 pages, 7 figures

2404.16020 2025-04-10 cs.CL 70%

Investigating Adversarial Trigger Transfer in Large Language Models

Nicholas Meade, Arkil Patel, Siva Reddy

专题命中偏好对齐：safety(abstract)；jailbreak(abstract)；分类 cs.CL

1. 偏好对齐 3273 篇

VARCO-VISION-2.0 Technical Report

Activation Steering Meets Preference Optimization: Defense Against Jailbreaks in Vision Language Models

HEAL: A Hypothesis-Based Preference-Aware Analysis Framework

InfiAlign: A Scalable and Sample-Efficient Framework for Aligning LLMs to Enhance Reasoning Capabilities

Auto-TA: Towards Scalable Automated Thematic Analysis (TA) via Multi-Agent Large Language Models with Reinforcement Learning

Towards Hallucination-Free Music: A Reinforcement Learning Preference Optimization Framework for Reliable Song Generation

A Survey on Data Security in Large Language Models

Bridging the Gap in Vision Language Models in Identifying Unsafe Concepts Across Modalities

Extragradient Preference Optimization (EGPO): Beyond Last-Iterate Convergence for Nash Learning from Human Feedback

Finding the Sweet Spot: Preference Data Construction for Scaling Preference Optimization

VICCA: Visual Interpretation and Comprehension of Chest X-ray Anomalies in Generated Report Without Human Feedback

Sailing by the Stars: A Survey on Reward Models and Learning Strategies for Learning from Rewards

Explicit Preference Optimization: No Need for an Implicit Reward Model

Aligning Large Language Models with Implicit Preferences from User-Generated Content

AutoMixAlign: Adaptive Data Mixing for Multi-Task Preference Optimization in LLMs

Curiosity-Driven Reinforcement Learning from Human Feedback

Hybrid Preferences: Learning to Route Instances for Human vs. AI Feedback

Intuitionistic Fuzzy Sets for Large Language Model Data Annotation: A Novel Approach to Side-by-Side Preference Labeling

ExpandR: Teaching Dense Retrievers Beyond Queries with LLM Guidance

LPOI: Listwise Preference Optimization for Vision Language Models

ASPO: Adaptive Sentence-Level Preference Optimization for Fine-Grained Multimodal Reasoning

Reinforcement Learning from User Feedback

No Preference Left Behind: Group Distributional Preference Optimization

The Root Shapes the Fruit: On the Persistence of Gender-Exclusive Harms in Aligned Language Models

Latent Preference Coding: Aligning Large Language Models via Discrete Latent Codes

SIMPLEMIX: Frustratingly Simple Mixing of Off- and On-policy Data in Language Model Preference Learning

Learning Explainable Dense Reward Shapes via Bayesian Optimization

A Self-Supervised Reinforcement Learning Approach for Fine-Tuning Large Language Models Using Cross-Attention Signals

Adversarial Training of Reward Models

Investigating Adversarial Trigger Transfer in Large Language Models