大模型对齐与安全

Few-shot Steerable Alignment: Adapting Rewards and LLM Policies with Neural Processes

Katarzyna Kobalczyk, Claudio Fanconi, Hao Sun, Mihaela van der Schaar

专题命中 偏好对齐 ：alignment(title,abstract)；分类 cs.AI、cs.LG

Does RLHF Scale? Exploring the Impacts From Data, Model, and Method

Zhenyu Hou, Pengfan Du, Yilin Niu, Zhengxiao Du, Aohan Zeng, Xiao Liu, Minlie Huang, Hongning Wang, Jie Tang, Yuxiao Dong

专题命中 偏好对齐 ：RLHF(title,abstract)；分类 cs.CL、cs.LG

ALMA: Alignment with Minimal Annotation

Michihiro Yasunaga, Leonid Shamis, Chunting Zhou, Andrew Cohen, Jason Weston, Luke Zettlemoyer, Marjan Ghazvininejad

专题命中 偏好对齐 ：alignment(title,abstract)；分类 cs.CL、cs.LG

Word Alignment as Preference for Machine Translation

Qiyu Wu, Masaaki Nagata, Zhongtao Miao, Yoshimasa Tsuruoka

专题命中 偏好对齐 ：alignment(title,abstract)；分类 cs.CL、cs.AI

Comments EMNLP 2024 Main

Value Imprint: A Technique for Auditing the Human Values Embedded in RLHF Datasets

Ike Obi, Rohan Pant, Srishti Shekhar Agrawal, Maham Ghazanfar, Aaron Basiletti

专题命中 偏好对齐 ：RLHF(title,abstract)；分类 cs.AI、cs.LG

Axioms for AI Alignment from Human Feedback

Luise Ge, Daniel Halpern, Evi Micha, Ariel D. Procaccia, Itai Shapira, Yevgeniy Vorobeychik, Junlin Wu

专题命中 偏好对齐 ：alignment(title)；RLHF(abstract)；分类 cs.AI、cs.LG

InfoRM: Mitigating Reward Hacking in RLHF via Information-Theoretic Reward Modeling

Yuchun Miao, Sen Zhang, Liang Ding, Rong Bao, Lefei Zhang, Dacheng Tao

专题命中 偏好对齐 ：RLHF(title,abstract)；分类 cs.AI、cs.LG

Comments The paper has been accepted by NeurIPS 2024

Reward Difference Optimization For Sample Reweighting In Offline RLHF

Shiqi Wang, Zhengze Zhang, Rui Zhao, Fei Tan, Cam Tu Nguyen

专题命中 偏好对齐 ：RLHF(title,abstract)；分类 cs.CL、cs.AI

Comments EMNLP 2024 findings

Flow-DPO: Improving LLM Mathematical Reasoning through Online Multi-Agent Learning

Yihe Deng, Paul Mineiro

专题命中 偏好对齐 ：DPO(title,abstract)；分类 cs.CL、cs.LG

Comments 5 pages, 4 figures, 1 table

Inference time LLM alignment in single and multidomain preference spectrum

Sadat Shahriar, Zheng Qi, Nikolaos Pappas, Srikanth Doss, Monica Sunkara, Kishaloy Halder, Manuel Mager, Yassine Benajiba

专题命中 偏好对齐 ：alignment(title,abstract)；分类 cs.CL、cs.LG

LOGO -- Long cOntext aliGnment via efficient preference Optimization

Zecheng Tang, Zechen Sun, Juntao Li, Qiaoming Zhu, Min Zhang

专题命中 偏好对齐 ：alignment(title,abstract)；分类 cs.CL、cs.AI

Boosting Deductive Reasoning with Step Signals In RLHF

Jialian Li, Yipin Zhang, Wei Shen, Yuzi Yan, Jian Xie, Dong Yan

专题命中 偏好对齐 ：RLHF(title,abstract)；分类 cs.AI、cs.LG

FlipGuard: Defending Preference Alignment against Update Regression with Constrained Optimization

Mingye Zhu, Yi Liu, Quan Wang, Junbo Guo, Zhendong Mao

专题命中 偏好对齐 ：alignment(title,abstract)；分类 cs.CL、cs.AI

Comments Accepted by EMNLP 2024 Main track

$β$-DPO: Direct Preference Optimization with Dynamic $β$

Junkang Wu, Yuexiang Xie, Zhengyi Yang, Jiancan Wu, Jinyang Gao, Bolin Ding, Xiang Wang, Xiangnan He

专题命中 偏好对齐 ：DPO(title,abstract)；分类 cs.AI、cs.LG

Comments NeurIPS 2024

Magpie: Alignment Data Synthesis from Scratch by Prompting Aligned LLMs with Nothing

Zhangchen Xu, Fengqing Jiang, Luyao Niu, Yuntian Deng, Radha Poovendran, Yejin Choi, Bill Yuchen Lin

专题命中 偏好对齐 ：alignment(title,abstract)；分类 cs.CL、cs.AI

Comments Link: https://magpie-align.github.io/

Investigating on RLHF methodology

Alexey Kutalev, Sergei Markoff

专题命中 偏好对齐 ：RLHF(title)；alignment(abstract)；分类 cs.AI、cs.LG

Comments 23 pages, 6 figures, 6 tables

Systematic Characterization of the Effectiveness of Alignment in Large Language Models for Categorical Decisions

Isaac Kohane

专题命中 偏好对齐 ：alignment(title,abstract)；分类 cs.CL、cs.AI

Comments 19 pages (without Appendix) Appendix 7 pages. 7 Figures

Value Alignment from Unstructured Text

Inkit Padhi, Karthikeyan Natesan Ramamurthy, Prasanna Sattigeri, Manish Nagireddy, Pierre Dognin, Kush R. Varshney

专题命中 偏好对齐 ：alignment(title,abstract)；分类 cs.CL、cs.LG

Strong and weak alignment of large language models with human values

Mehdi Khamassi, Marceau Nahon, Raja Chatila

专题命中 偏好对齐 ：alignment(title,abstract)；分类 cs.CL、cs.AI

Comments Accepted for publication in Scientific Reports, special issue on AI aligment

Understanding the Learning Dynamics of Alignment with Human Feedback

Shawn Im, Yixuan Li

专题命中 偏好对齐 ：alignment(title,abstract)；分类 cs.AI、cs.LG

A Long Way to Go: Investigating Length Correlations in RLHF

Prasann Singhal, Tanya Goyal, Jiacheng Xu, Greg Durrett

专题命中 偏好对齐 ：RLHF(title,abstract)；分类 cs.CL、cs.LG

Comments 21 pages, 13 figures, Accepted to COLM 2024

Prototypical Reward Network for Data-Efficient RLHF

Jinghan Zhang, Xiting Wang, Yiqiao Jin, Changyu Chen, Xinhao Zhang, Kunpeng Liu

专题命中 偏好对齐 ：RLHF(title,abstract)；分类 cs.CL、cs.AI

Comments Accepted by ACL 2024

LANE: Logic Alignment of Non-tuning Large Language Models and Online Recommendation Systems for Explainable Reason Generation

Hongke Zhao, Songming Zheng, Likang Wu, Bowen Yu, Jing Wang

专题命中 偏好对齐 ：alignment(title,abstract)；分类 cs.CL、cs.LG

Knowledgeable Preference Alignment for LLMs in Domain-specific Question Answering

Yichi Zhang, Zhuo Chen, Yin Fang, Yanxi Lu, Fangming Li, Wen Zhang, Huajun Chen

专题命中 偏好对齐 ：alignment(title,abstract)；分类 cs.CL、cs.AI

Comments Accepted by ACL 2024 (Findings). Code is available at https://github.com/zjukg/KnowPAT

Group Robust Preference Optimization in Reward-free RLHF

Shyam Sundhar Ramesh, Yifan Hu, Iason Chaimalas, Viraj Mehta, Pier Giuseppe Sessa, Haitham Bou Ammar, Ilija Bogunovic

专题命中 偏好对齐 ：RLHF(title,abstract)；分类 cs.CL、cs.LG

Comments Preprint

Offline Regularised Reinforcement Learning for Large Language Models Alignment

Pierre Harvey Richemond, Yunhao Tang, Daniel Guo, Daniele Calandriello, Mohammad Gheshlaghi Azar, Rafael Rafailov, Bernardo Avila Pires, Eugene Tarassov, Lucas Spangher, Will Ellsworth, Aliaksei Severyn, Jonathan Mallinson, Lior Shani, Gil Shamir, Rishabh Joshi, Tianqi Liu, Remi Munos, Bilal Piot

专题命中 偏好对齐 ：alignment(title,abstract)；分类 cs.AI、cs.LG

AI Alignment with Changing and Influenceable Reward Functions

Micah Carroll, Davis Foote, Anand Siththaranjan, Stuart Russell, Anca Dragan

专题命中 偏好对齐 ：alignment(title,abstract)；分类 cs.AI、cs.LG

Comments Accepted to ICML 2024

RLHF from Heterogeneous Feedback via Personalization and Preference Aggregation

Chanwoo Park, Mingyang Liu, Dingwen Kong, Kaiqing Zhang, Asuman Ozdaglar

专题命中 偏好对齐 ：RLHF(title,abstract)；分类 cs.AI、cs.LG

Comments Added experiments

Adversarial DPO: Harnessing Harmful Data for Reducing Toxicity with Minimal Impact on Coherence and Evasiveness in Dialogue Agents

San Kim, Gary Geunbae Lee

专题命中 偏好对齐 ：DPO(title,abstract)；分类 cs.CL、cs.AI

Comments 15 pages, 7 figures, accepted to NAACL findings 2024

Provably Robust DPO: Aligning Language Models with Noisy Feedback

Sayak Ray Chowdhury, Anush Kini, Nagarajan Natarajan

专题命中 偏好对齐 ：DPO(title,abstract)；分类 cs.CL、cs.LG