大模型对齐与安全 - arXivDaily 专题

2412.17696 2025-03-28 cs.CL 83%

Understanding the Logic of Direct Preference Alignment through Logic

Kyle Richardson, Vivek Srikumar, Ashish Sabharwal

专题命中偏好对齐：alignment(title,abstract)；DPO(abstract)；分类 cs.CL

2503.18454 2025-03-25 cs.CV cs.LG 83%

InPO: Inversion Preference Optimization with Reparametrized DDIM for Efficient Diffusion Model Alignment

Yunhong Lu, Qichao Wang, Hengyuan Cao, Xierui Wang, Xiaoyin Xu, Min Zhang

专题命中偏好对齐：alignment(title,abstract)；DPO(abstract)；分类 cs.LG

Comments Accepted by CVPR2025

2408.11779 2025-03-11 cs.CL 83%

Personality Alignment of Large Language Models

Minjun Zhu, Yixuan Weng, Linyi Yang, Yue Zhang

专题命中偏好对齐：alignment(title,abstract)；DPO(abstract)；分类 cs.CL

Comments Acecpt in ICLR 2025

2410.02451 2025-03-11 cs.AI 83%

Strong Preferences Affect the Robustness of Preference Models and Value Alignment

Ziwei Xu, Mohan Kankanhalli

专题命中偏好对齐：alignment(title,abstract)；safety(abstract)；分类 cs.AI

Comments 21 Pages. Accepted by ICLR 2025

2503.05079 2025-03-10 cs.LG 83%

On a Connection Between Imitation Learning and RLHF

Teng Xiao, Yige Yuan, Mingxiao Li, Zhengyu Chen, Vasant G Honavar

专题命中偏好对齐：RLHF(title,abstract)；alignment(abstract)；分类 cs.LG

Comments ICLR 2025

2410.09724 2025-03-04 cs.CL 83%

Taming Overconfidence in LLMs: Reward Calibration in RLHF

Jixuan Leng, Chengsong Huang, Banghua Zhu, Jiaxin Huang

专题命中偏好对齐：RLHF(title,abstract)；alignment(abstract)；分类 cs.CL

2411.10914 2025-02-21 cs.CL 83%

BPO: Towards Balanced Preference Optimization between Knowledge Breadth and Depth in Alignment

Sizhe Wang, Yongqi Tong, Hengyuan Zhang, Dawei Li, Xin Zhang, Tianlong Chen

专题命中偏好对齐：alignment(title,abstract)；RLHF(abstract)；分类 cs.CL

Comments The 2025 Annual Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics (NAACL 2025)- Main Conference

2501.12735 2025-02-10 cs.LG 83%

Online Preference Alignment for Language Models via Count-based Exploration

Chenjia Bai, Yang Zhang, Shuang Qiu, Qiaosheng Zhang, Kang Xu, Xuelong Li

专题命中偏好对齐：alignment(title,abstract)；RLHF(abstract)；分类 cs.LG

Comments Accepted by ICLR 2025

2501.15453 2025-01-29 cs.CL 83%

Data-adaptive Safety Rules for Training Reward Models

Xiaomin Li, Mingye Gao, Zhiwei Zhang, Jingxuan Fan, Weiyu Li

专题命中偏好对齐：safety(title,abstract)；RLHF(abstract)；分类 cs.CL

2501.12895 2025-01-23 cs.CL 83%

Test-Time Preference Optimization: On-the-Fly Alignment via Iterative Textual Feedback

Yafu Li, Xuyang Hu, Xiaoye Qu, Linjie Li, Yu Cheng

专题命中偏好对齐：alignment(title,abstract)；safety(abstract)；分类 cs.CL

Comments 43 pages; work in progress

2412.15623 2024-12-23 cs.CR cs.AI 83%

JailPO: A Novel Black-box Jailbreak Framework via Preference Optimization against Aligned LLMs

Hongyi Li, Jiawei Ye, Jie Wu, Tianjie Yan, Chu Wang, Zhixin Li

专题命中偏好对齐：jailbreak(title,abstract)；alignment(abstract)；分类 cs.AI

Comments Accepted by AAAI 2025

2411.08733 2024-11-15 cs.CL 83%

Dynamic Rewarding with Prompt Optimization Enables Tuning-free Self-Alignment of Language Models

Somanshu Singla, Zhen Wang, Tianyang Liu, Abdullah Ashfaq, Zhiting Hu, Eric P. Xing

专题命中偏好对齐：alignment(title,abstract)；RLHF(abstract)；分类 cs.CL

Comments EMNLP 2024 Main

2411.01245 2024-11-05 cs.CL 83%

PMoL: Parameter Efficient MoE for Preference Mixing of LLM Alignment

Dongxu Liu, Bing Xu, Yinzhuo Chen, Bufan Xu, Wenpeng Lu, Muyun Yang, Tiejun Zhao

专题命中偏好对齐：alignment(title,abstract)；RLHF(abstract)；分类 cs.CL

2405.17888 2024-10-29 cs.AI 83%

Getting More Juice Out of the SFT Data: Reward Learning from Human Demonstration Improves SFT for LLM Alignment

Jiaxiang Li, Siliang Zeng, Hoi-To Wai, Chenliang Li, Alfredo Garcia, Mingyi Hong

专题命中偏好对齐：alignment(title,abstract)；RLHF(abstract)；分类 cs.AI

2410.09421 2024-10-21 cs.CV cs.CL 83%

VLFeedback: A Large-Scale AI Feedback Dataset for Large Vision-Language Models Alignment

Lei Li, Zhihui Xie, Mukai Li, Shunian Chen, Peiyi Wang, Liang Chen, Yazheng Yang, Benyou Wang, Lingpeng Kong, Qi Liu

专题命中偏好对齐：alignment(title,abstract)；safety(abstract)；分类 cs.CL

Comments EMNLP 2024 Main Conference camera-ready version (fixed small typos). This article supersedes arXiv:2312.10665

2405.12739 2024-10-14 cs.LG 83%

SPO: Multi-Dimensional Preference Sequential Alignment With Implicit Reward Modeling

Xingzhou Lou, Junge Zhang, Jian Xie, Lifeng Liu, Dong Yan, Kaiqi Huang

专题命中偏好对齐：alignment(title,abstract)；harmlessness(abstract)；分类 cs.LG

2409.19256 2024-10-03 cs.LG cs.DC 83%

HybridFlow: A Flexible and Efficient RLHF Framework

Guangming Sheng, Chi Zhang, Zilingfeng Ye, Xibin Wu, Wang Zhang, Ru Zhang, Yanghua Peng, Haibin Lin, Chuan Wu

专题命中偏好对齐：RLHF(title,abstract)；alignment(abstract)；分类 cs.LG

2405.01511 2024-08-08 cs.CL 83%

D2PO: Discriminator-Guided DPO with Response Evaluation Models

Prasann Singhal, Nathan Lambert, Scott Niekum, Tanya Goyal, Greg Durrett

专题命中偏好对齐：DPO(title,abstract)；RLHF(abstract)；分类 cs.CL

Comments 20 pages, 12 figures, Accepted to COLM 2024

2407.03051 2024-07-19 cs.CL 83%

Improving Conversational Abilities of Quantized Large Language Models via Direct Preference Alignment

Janghwan Lee, Seongmin Park, Sukjin Hong, Minsoo Kim, Du-Seong Chang, Jungwook Choi

专题命中偏好对齐：alignment(title,abstract)；RLHF(abstract)；分类 cs.CL

Comments ACL 2024 Main

2407.00782 2024-07-16 cs.CL 83%

Step-Controlled DPO: Leveraging Stepwise Error for Enhanced Mathematical Reasoning

Zimu Lu, Aojun Zhou, Ke Wang, Houxing Ren, Weikang Shi, Junting Pan, Mingjie Zhan, Hongsheng Li

专题命中偏好对齐：DPO(title,abstract)；alignment(abstract)；分类 cs.CL

2407.02477 2024-07-03 cs.CV cs.CL 83%

Understanding Alignment in Multimodal LLMs: A Comprehensive Study

Elmira Amirloo, Jean-Philippe Fauconnier, Christoph Roesmann, Christian Kerl, Rinu Boney, Yusu Qian, Zirui Wang, Afshin Dehghan, Yinfei Yang, Zhe Gan, Peter Grasch

专题命中偏好对齐：alignment(title,abstract)；DPO(abstract)；分类 cs.CL

2401.11458 2024-07-03 cs.CL 83%

Linear Alignment: A Closed-form Solution for Aligning Human Preferences without Tuning and Feedback

Songyang Gao, Qiming Ge, Wei Shen, Shihan Dou, Junjie Ye, Xiao Wang, Rui Zheng, Yicheng Zou, Zhi Chen, Hang Yan, Qi Zhang, Dahua Lin

专题命中偏好对齐：alignment(title,abstract)；RLHF(abstract)；分类 cs.CL

Comments Accepted by ICML2024, I'm still preparing a better vision

2406.19188 2024-06-28 cs.LG 83%

Averaging log-likelihoods in direct alignment

Nathan Grinsztajn, Yannis Flet-Berliac, Mohammad Gheshlaghi Azar, Florian Strub, Bill Wu, Eugene Choi, Chris Cremer, Arash Ahmadian, Yash Chandak, Olivier Pietquin, Matthieu Geist

专题命中偏好对齐：alignment(title,abstract)；RLHF(abstract)；分类 cs.LG

2406.16316 2024-06-25 cs.CL cs.AI cs.CY cs.LG 83%

Does Cross-Cultural Alignment Change the Commonsense Morality of Language Models?

Yuu Jinnai

专题命中偏好对齐：alignment(title,abstract)；分类 cs.CL、cs.AI、cs.CY

Comments The 2nd Workshop on Cross-Cultural Considerations in NLP (C3NLP) at ACL 2024

2406.08842 2024-06-14 cs.CL 83%

ContraSolver: Self-Alignment of Language Models by Resolving Internal Preference Contradictions

Xu Zhang, Xunjian Yin, Xiaojun Wan

专题命中偏好对齐：alignment(title,abstract)；DPO(abstract)；分类 cs.CL

2402.00856 2024-06-06 cs.CL 83%

Towards Efficient Exact Optimization of Language Model Alignment

Haozhe Ji, Cheng Lu, Yilin Niu, Pei Ke, Hongning Wang, Jun Zhu, Jie Tang, Minlie Huang

专题命中偏好对齐：alignment(title,abstract)；DPO(abstract)；分类 cs.CL

Comments 24 pages, 9 figures

Journal ref Forty-first International Conference on Machine Learning (ICML 2024)

2404.10271 2024-06-05 cs.LG cs.AI cs.CL cs.CY cs.GT 83%

Social Choice Should Guide AI Alignment in Dealing with Diverse Human Feedback

Vincent Conitzer, Rachel Freedman, Jobst Heitzig, Wesley H. Holliday, Bob M. Jacobs, Nathan Lambert, Milan Mossé, Eric Pacuit, Stuart Russell, Hailey Schoelkopf, Emanuel Tewolde, William S. Zwicker

专题命中偏好对齐：alignment(title)；safety(abstract)；分类 cs.CL、cs.AI、cs.CY

Comments 15 pages, 4 figures

2405.20335 2024-05-31 cs.CL 83%

Xwin-LM: Strong and Scalable Alignment Practice for LLMs

Bolin Ni, JingCheng Hu, Yixuan Wei, Houwen Peng, Zheng Zhang, Gaofeng Meng, Han Hu

专题命中偏好对齐：alignment(title,abstract)；DPO(abstract)；分类 cs.CL

2405.18718 2024-05-30 cs.CL 83%

Efficient Model-agnostic Alignment via Bayesian Persuasion

Fengshuo Bai, Mingzhi Wang, Zhaowei Zhang, Boyuan Chen, Yinda Xu, Ying Wen, Yaodong Yang

专题命中偏好对齐：alignment(title,abstract)；RLHF(abstract)；分类 cs.CL

2311.04072 2024-04-16 cs.CL 83%

Beyond Imitation: Leveraging Fine-grained Quality Signals for Alignment

Geyang Guo, Ranchi Zhao, Tianyi Tang, Wayne Xin Zhao, Ji-Rong Wen

专题命中偏好对齐：alignment(title,abstract)；RLHF(abstract)；分类 cs.CL

1. 偏好对齐 3261 篇

Understanding the Logic of Direct Preference Alignment through Logic

InPO: Inversion Preference Optimization with Reparametrized DDIM for Efficient Diffusion Model Alignment

Personality Alignment of Large Language Models

Strong Preferences Affect the Robustness of Preference Models and Value Alignment

On a Connection Between Imitation Learning and RLHF

Taming Overconfidence in LLMs: Reward Calibration in RLHF

BPO: Towards Balanced Preference Optimization between Knowledge Breadth and Depth in Alignment

Online Preference Alignment for Language Models via Count-based Exploration

Data-adaptive Safety Rules for Training Reward Models

Test-Time Preference Optimization: On-the-Fly Alignment via Iterative Textual Feedback

JailPO: A Novel Black-box Jailbreak Framework via Preference Optimization against Aligned LLMs

Dynamic Rewarding with Prompt Optimization Enables Tuning-free Self-Alignment of Language Models

PMoL: Parameter Efficient MoE for Preference Mixing of LLM Alignment

Getting More Juice Out of the SFT Data: Reward Learning from Human Demonstration Improves SFT for LLM Alignment

VLFeedback: A Large-Scale AI Feedback Dataset for Large Vision-Language Models Alignment

SPO: Multi-Dimensional Preference Sequential Alignment With Implicit Reward Modeling

HybridFlow: A Flexible and Efficient RLHF Framework

D2PO: Discriminator-Guided DPO with Response Evaluation Models

Improving Conversational Abilities of Quantized Large Language Models via Direct Preference Alignment

Step-Controlled DPO: Leveraging Stepwise Error for Enhanced Mathematical Reasoning

Understanding Alignment in Multimodal LLMs: A Comprehensive Study

Linear Alignment: A Closed-form Solution for Aligning Human Preferences without Tuning and Feedback

Averaging log-likelihoods in direct alignment

Does Cross-Cultural Alignment Change the Commonsense Morality of Language Models?

ContraSolver: Self-Alignment of Language Models by Resolving Internal Preference Contradictions

Towards Efficient Exact Optimization of Language Model Alignment

Social Choice Should Guide AI Alignment in Dealing with Diverse Human Feedback

Xwin-LM: Strong and Scalable Alignment Practice for LLMs

Efficient Model-agnostic Alignment via Bayesian Persuasion

Beyond Imitation: Leveraging Fine-grained Quality Signals for Alignment