大模型对齐与安全 - arXivDaily 专题

2510.00647 2025-10-02 cs.CL 79%

MCM-DPO: Multifaceted Cross-Modal Direct Preference Optimization for Alt-text Generation

Jinlan Fu, Shenzhen Huangfu, Hao Fei, Yichong Huang, Xiaoyu Shen, Xipeng Qiu, See-Kiong Ng

机构 * National University of Singapore（新加坡国立大学）； Fudan University（复旦大学）； Harbin Institute of Technology（哈尔滨工业大学）； Eastern Institute of Technology（东方技术研究所）

专题命中偏好对齐：DPO(title,abstract)；分类 cs.CL

Comments Accepted by ACM MM 2025

2509.23667 2025-09-30 cs.LG 79%

Why Alignment Must Precede Distillation: A Minimal Working Explanation

Sungmin Cha, Kyunghyun Cho

机构 * New York University（纽约大学）； Genentech（基因泰克）

专题命中偏好对齐：alignment(title,abstract)；分类 cs.LG

Comments Preprint

2501.04561 2025-09-24 cs.CL cs.CV 79%

OpenOmni: Advancing Open-Source Omnimodal Large Language Models with Progressive Multimodal Alignment and Real-Time Self-Aware Emotional Speech Synthesis

Run Luo, Ting-En Lin, Haonan Zhang, Yuchuan Wu, Xiong Liu, Min Yang, Yongbin Li, Longze Chen, Jiaming Li, Lei Zhang, Xiaobo Xia, Hamid Alinejad-Rokny, Fei Huang

机构 * Shenzhen Key Laboratory for High Performance Data Mining（深圳高性能数据挖掘重点实验室）； Shenzhen Institute of Advanced Technology（深圳先进技术研究院）； Chinese Academy of Sciences（中国科学院）； University of Chinese Academy of Sciences（中国科学院大学）； Tongyi Laboratory（通义实验室）； University of New South Wales（新南威尔士大学）； National University of Singapore（新加坡国立大学）； University of Science and Technology of China（中国科学技术大学）； MoE Key Laboratory of Brain-inspired Intelligent Perception and Cognition（脑启发智能感知与认知重点实验室）

专题命中偏好对齐：alignment(title,abstract)；分类 cs.CL

2509.04713 2025-09-08 cs.LG 79%

Natural Spectral Fusion: p-Exponent Cyclic Scheduling and Early Decision-Boundary Alignment in First-Order Optimization

Gongyue Zhang, Honghai Liu

专题命中偏好对齐：alignment(title,abstract)；分类 cs.LG

2505.02666 2025-09-03 cs.CL 79%

A Survey on Progress in LLM Alignment from the Perspective of Reward Design

Miaomiao Ji, Yanqiu Wu, Zhibin Wu, Shoujin Wang, Jian Yang, Mark Dras, Usman Naseem

专题命中偏好对齐：alignment(title,abstract)；分类 cs.CL

Comments Preprint

2508.19532 2025-08-28 cs.CL 79%

Alignment with Fill-In-the-Middle for Enhancing Code Generation

Houxing Ren, Zimu Lu, Weikang Shi, Haotian Hou, Yunqiao Yang, Ke Wang, Aojun Zhou, Junting Pan, Mingjie Zhan, Hongsheng Li

机构 * CUHK MMLab（CUHK语音实验室）； SenseTime Research（商汤科技研究院）； CPII under InnoHK（创新香港下的CPII）； Shanghai AI Laboratory（上海人工智能实验室）； Beihang University（北航）

专题命中偏好对齐：alignment(title)；DPO(abstract)；分类 cs.CL

Comments Accepted to EMNLP 2025 (main conference)

2508.17718 2025-08-26 cs.CV cs.AI 79%

Instant Preference Alignment for Text-to-Image Diffusion Models

Yang Li, Songlin Yang, Xiaoxuan Han, Wei Wang, Jing Dong, Yueming Lyu, Ziyu Xue

机构 * New Laboratory of Pattern Recognition, CASIA（模式识别新实验室，中国科学院自动化研究所）； The Hong Kong University of Science and Technology（香港科技大学）； Nanjing university（南京大学）； Academy of Broadcasting Science, NRTA（广播科学研究院，国家广播电视总局）

专题命中偏好对齐：alignment(title,abstract)；分类 cs.AI

Comments 17 figures

2508.08550 2025-08-13 cs.SD cs.CL 79%

Fine-grained Video Dubbing Duration Alignment with Segment Supervised Preference Optimization

Chaoqun Cui, Liangbin Huang, Shijing Wang, Zhe Tong, Zhaolong Huang, Xiao Zeng, Xiaofeng Liu

机构 * Alibaba Digital Media and Entertainment Group（阿里巴巴数字媒体与娱乐集团）； School of Software Engineering, Huazhong University of Science and Technology（华中科技大学软件学院）； Beijing Key Laboratory of Traffic Data Mining and Embodied Intelligence, Beijing Jiaotong University（北京交通大学交通数据挖掘与具身智能重点实验室）

专题命中偏好对齐：alignment(title,abstract)；分类 cs.CL

Comments This paper is accepted by ACL2025 (Main)

Journal ref Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2025: 4524-4546

2508.04963 2025-08-08 cs.IR cs.LG 79%

A Metric for MLLM Alignment in Large-scale Recommendation

Yubin Zhang, Yanhua Huang, Haiming Xu, Mingliang Qi, Chang Wang, Jiarui Jin, Xiangyuan Ren, Xiaodan Wang, Ruiwen Xu

机构 * Xiaohongshu Inc.（小红书公司）

专题命中偏好对齐：alignment(title,abstract)；分类 cs.LG

Comments Pre-print.Under Review

2508.04698 2025-08-07 cs.CL 79%

FaST: Feature-aware Sampling and Tuning for Personalized Preference Alignment with Limited Data

Thibaut Thonet, Germán Kruszewski, Jos Rozen, Pierre Erbacher, Marc Dymetman

机构 * NAVER Labs Europe（NAVER欧洲实验室）； Independent Researcher（独立研究者）

专题命中偏好对齐：alignment(title,abstract)；分类 cs.CL

2503.12854 2025-07-29 cs.CL 79%

Enhancing LLM Reasoning with Iterative DPO: A Comprehensive Empirical Investigation

Songjun Tu, Jiahao Lin, Xiangyu Tian, Qichao Zhang, Linjing Li, Yuqian Fu, Nan Xu, Wei He, Xiangyuan Lan, Dongmei Jiang, Dongbin Zhao

机构 * Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）； Pengcheng Laboratory（鹏城实验室）； School of Artificial Intelligence, University of Chinese Academy of Sciences（中国科学院大学人工智能学院）； Wenge Technology（文生科技）； Fudan University（复旦大学）

专题命中偏好对齐：DPO(title,abstract)；分类 cs.CL

Comments 23pages

Journal ref COLM2025

2507.07939 2025-07-23 cs.CL 79%

SAGE: A Visual Language Model for Anomaly Detection via Fact Enhancement and Entropy-aware Alignment

Guoxin Zang, Xue Li, Donglin Di, Lanshun Nie, Dechen Zhan, Yang Song, Lei Fan

机构 * Harbin Institute of Technology（哈尔滨工业大学）； University of New South Wales（新南威尔士大学）

专题命中偏好对齐：alignment(title)；DPO(abstract)；分类 cs.CL

Comments Accepted by ACMMM2025

2507.09016 2025-07-17 cs.LG 79%

Enhancing RLHF with Human Gaze Modeling

Karim Galliamov, Ivan Titov, Ilya Pershin

专题命中偏好对齐：RLHF(title,abstract)；分类 cs.LG

2410.08193 2025-07-16 cs.CL 79%

GenARM: Reward Guided Generation with Autoregressive Reward Model for Test-time Alignment

Yuancheng Xu, Udari Madhushani Sehwag, Alec Koppel, Sicheng Zhu, Bang An, Furong Huang, Sumitra Ganesh

专题命中偏好对齐：alignment(title,abstract)；分类 cs.CL

Comments Published at the Thirteenth International Conference on Learning Representations (ICLR 2025)

2504.10208 2025-07-08 cs.IR cs.LG 79%

From Prompting to Alignment: A Generative Framework for Query Recommendation

Erxue Min, Hsiu-Yuan Huang, Xihong Yang, Min Yang, Xin Jia, Yunfang Wu, Hengyi Cai, Junfeng Wang, Shuaiqiang Wang, Dawei Yin

机构 * Baidu Inc.（百度公司）； National Key Laboratory for Multimedia Information Processing, Peking University（北京大学多媒体信息处理国家重点实验室）； Chinese Academy of Sciences（中国科学院）

专题命中偏好对齐：alignment(title,abstract)；分类 cs.LG

2507.02173 2025-07-04 cs.AI 79%

Data Diversification Methods In Alignment Enhance Math Performance In LLMs

Berkan Dokmeci, Qingyang Wu, Ben Athiwaratkun, Ce Zhang, Shuaiwen Leon Song, James Zou

机构 * Washington University in St. Louis（华盛顿大学圣路易斯分校）； Together AI ； University of Chicago（芝加哥大学）； Stanford University（斯坦福大学）

专题命中偏好对齐：alignment(title,abstract)；分类 cs.AI

2506.12446 2025-07-01 cs.CL 79%

From Outcomes to Processes: Guiding PRM Learning from ORM for Inference-Time Alignment

Bin Xie, Bingbing Xu, Yige Yuan, Shengmao Zhu, Huawei Shen

机构 * State Key Laboratory of AI Safety, Institute of Computing Technology, CAS（人工智能安全国家重点实验室、计算技术研究所、中国科学院）

专题命中偏好对齐：alignment(title,abstract)；分类 cs.CL

2410.03145 2025-07-01 cs.CL 79%

Margin Matching Preference Optimization: Enhanced Model Alignment with Granular Feedback

Kyuyoung Kim, Ah Jeong Seo, Hao Liu, Jinwoo Shin, Kimin Lee

机构 * KAIST（韩国科学技术院）； UC Berkeley（加州大学伯克利分校）

专题命中偏好对齐：alignment(title,abstract)；分类 cs.CL

Comments EMNLP 2024 Findings

2506.18245 2025-06-24 cs.CR cs.AI cs.SE 79%

Smart-LLaMA-DPO: Reinforced Large Language Model for Explainable Smart Contract Vulnerability Detection

Lei Yu, Zhirong Huang, Hang Yuan, Shiqi Cheng, Li Yang, Fengjun Zhang, Chenjie Shen, Jiajia Ma, Jingyuan Zhang, Junyi Lu, Chun Zuo

机构 * Institute of Software, Chinese Academy of Sciences（中国科学院软件研究所）； Sinosoft Company Limited（软通公司）

专题命中偏好对齐：DPO(title,abstract)；分类 cs.AI

Comments Accepted to ISSTA 2025

2506.17951 2025-06-24 cs.CL 79%

A Comprehensive Graph Framework for Question Answering with Mode-Seeking Preference Alignment

Quanwei Tang, Sophia Yat Mei Lee, Junshuang Wu, Dong Zhang, Shoushan Li, Erik Cambria, Guodong Zhou

专题命中偏好对齐：alignment(title,abstract)；分类 cs.CL

Comments acl 2025 findings

2506.17834 2025-06-24 cs.AI cs.HC 79%

Reflective Verbal Reward Design for Pluralistic Alignment

Carter Blair, Kate Larson, Edith Law

机构 * University of Waterloo（滑铁卢大学）

专题命中偏好对齐：alignment(title)；RLHF(abstract)；分类 cs.AI

Comments 9 pages, 3 figures, accepted to the IJCAI 2025 Human-Centred AI track. Project repository at: https://osf.io/8yxf2/

2502.07599 2025-06-09 cs.CL 79%

DPO-Shift: Shifting the Distribution of Direct Preference Optimization

Xiliang Yang, Feng Jiang, Qianen Zhang, Lei Zhao, Xiao Li

机构 * David S. Hippocampus Department of Computer Science（计算机科学系）

专题命中偏好对齐：DPO(title,abstract)；分类 cs.CL

2506.02460 2025-06-04 cs.CL 79%

MidPO: Dual Preference Optimization for Safety and Helpfulness in Large Language Models via a Mixture of Experts Framework

Yupeng Qi, Ziyu Lyu, Min Yang, Yanlin Wang, Lu Bai, Lixin Cui

机构 * Sun Yat-sen University（中山大学）； Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences（深圳先进技术研究院、中国科学院）； Beijing Normal University（北京师范大学）； Central University of Finance and Economics（中央财经大学）

专题命中偏好对齐：safety(title,abstract)；分类 cs.CL

2501.19358 2025-06-03 cs.LG 79%

The Energy Loss Phenomenon in RLHF: A New Perspective on Mitigating Reward Hacking

Yuchun Miao, Sen Zhang, Liang Ding, Yuqi Zhang, Lefei Zhang, Dacheng Tao

专题命中偏好对齐：RLHF(title,abstract)；分类 cs.LG

Comments The paper has been accepted by ICML 2025

2505.23927 2025-06-02 cs.LG 79%

Thompson Sampling in Online RLHF with General Function Approximation

Songtao Feng, Jie Fu

机构 * Department of Electrical and Computer Engineering, University of Florida（电气与计算机工程系，佛罗里达大学）

专题命中偏好对齐：RLHF(title,abstract)；分类 cs.LG

2505.21395 2025-05-28 cs.LG 79%

Square$χ$PO: Differentially Private and Robust $χ^2$-Preference Optimization in Offline Direct Alignment

Xingyu Zhou, Yulian Wu, Wenqian Weng, Francesco Orabona

专题命中偏好对齐：alignment(title,abstract)；分类 cs.LG

2505.20871 2025-05-28 cs.CL 79%

Divide-Then-Align: Honest Alignment based on the Knowledge Boundary of RAG

Xin Sun, Jianan Xie, Zhongqi Chen, Qiang Liu, Shu Wu, Yuehe Chen, Bowen Song, Weiqiang Wang, Zilei Wang, Liang Wang

专题命中偏好对齐：alignment(title)；DPO(abstract)；分类 cs.CL

Comments ACL 2025 main

2505.20556 2025-05-28 cs.LG 79%

Learning a Pessimistic Reward Model in RLHF

Yinglun Xu, Hangoo Kang, Tarun Suresh, Yuxuan Wan, Gagandeep Singh

机构 * University of Illinois Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）

专题命中偏好对齐：RLHF(title,abstract)；分类 cs.LG

2505.00038 2025-05-21 cs.CL 79%

HyPerAlign: Interpretable Personalized LLM Alignment via Hypothesis Generation

Cristina Garbacea, Chenhao Tan

机构 * University of Chicago（芝加哥大学）

专题命中偏好对齐：alignment(title,abstract)；分类 cs.CL

2505.12845 2025-05-20 cs.AI 79%

Multi-Level Aware Preference Learning: Enhancing RLHF for Complex Multi-Instruction Tasks

Ruopei Sun, Jianfeng Cai, Jinhua Zhu, Kangwen Zhao, Dongyun Xue, Wengang Zhou, Li Li, Houqiang Li

机构 * University of Science and Technology of China（中国科学技术大学）

专题命中偏好对齐：RLHF(title,abstract)；分类 cs.AI

1. 偏好对齐 3266 篇

MCM-DPO: Multifaceted Cross-Modal Direct Preference Optimization for Alt-text Generation

Why Alignment Must Precede Distillation: A Minimal Working Explanation

OpenOmni: Advancing Open-Source Omnimodal Large Language Models with Progressive Multimodal Alignment and Real-Time Self-Aware Emotional Speech Synthesis

Natural Spectral Fusion: p-Exponent Cyclic Scheduling and Early Decision-Boundary Alignment in First-Order Optimization

A Survey on Progress in LLM Alignment from the Perspective of Reward Design

Alignment with Fill-In-the-Middle for Enhancing Code Generation

Instant Preference Alignment for Text-to-Image Diffusion Models

Fine-grained Video Dubbing Duration Alignment with Segment Supervised Preference Optimization

A Metric for MLLM Alignment in Large-scale Recommendation

FaST: Feature-aware Sampling and Tuning for Personalized Preference Alignment with Limited Data

Enhancing LLM Reasoning with Iterative DPO: A Comprehensive Empirical Investigation

SAGE: A Visual Language Model for Anomaly Detection via Fact Enhancement and Entropy-aware Alignment

Enhancing RLHF with Human Gaze Modeling

GenARM: Reward Guided Generation with Autoregressive Reward Model for Test-time Alignment

From Prompting to Alignment: A Generative Framework for Query Recommendation

Data Diversification Methods In Alignment Enhance Math Performance In LLMs

From Outcomes to Processes: Guiding PRM Learning from ORM for Inference-Time Alignment

Margin Matching Preference Optimization: Enhanced Model Alignment with Granular Feedback

Smart-LLaMA-DPO: Reinforced Large Language Model for Explainable Smart Contract Vulnerability Detection

A Comprehensive Graph Framework for Question Answering with Mode-Seeking Preference Alignment

Reflective Verbal Reward Design for Pluralistic Alignment

DPO-Shift: Shifting the Distribution of Direct Preference Optimization

MidPO: Dual Preference Optimization for Safety and Helpfulness in Large Language Models via a Mixture of Experts Framework

The Energy Loss Phenomenon in RLHF: A New Perspective on Mitigating Reward Hacking

Thompson Sampling in Online RLHF with General Function Approximation

Square$χ$PO: Differentially Private and Robust $χ^2$-Preference Optimization in Offline Direct Alignment

Divide-Then-Align: Honest Alignment based on the Knowledge Boundary of RAG

Learning a Pessimistic Reward Model in RLHF

HyPerAlign: Interpretable Personalized LLM Alignment via Hypothesis Generation

Multi-Level Aware Preference Learning: Enhancing RLHF for Complex Multi-Instruction Tasks