大模型对齐与安全

DSPO: Direct Semantic Preference Optimization for Real-World Image Super-Resolution

Miaomiao Cai, Simiao Li, Wei Li, Xudong Huang, Hanting Chen, Jie Hu, Yunhe Wang

机构 * University of Science and Technology of China（中国科学技术大学） ； Huawei Noah’s Ark Lab（华为诺亚实验室） 纠错

专题命中 偏好对齐 ：alignment(abstract)；DPO(abstract)

LLMs Meet Finance: Fine-Tuning Foundation Models for the Open FinLLM Leaderboard

Varun Rao, Youran Sun, Mahendra Kumar, Tejas Mutneja, Agastya Mukherjee, Haizhao Yang

专题命中 偏好对齐 ：DPO(abstract)；分类 cs.CL、cs.AI、cs.LG

LLM Unlearning Reveals a Stronger-Than-Expected Coreset Effect in Current Benchmarks

Soumyadeep Pal, Changsheng Wang, James Diffenderfer, Bhavya Kailkhura, Sijia Liu

专题命中 偏好对齐 ：safety(abstract)；分类 cs.CL、cs.AI、cs.LG

PaMi-VDPO: Mitigating Video Hallucinations by Prompt-Aware Multi-Instance Video Preference Learning

Xinpeng Ding, Kui Zhang, Jianhua Han, Lanqing Hong, Hang Xu, Xiaomeng Li

专题命中 偏好对齐 ：alignment(abstract)；DPO(abstract)

Direct Preference Optimization for LLM-Enhanced Recommendation Systems

Chao Sun, Yaobo Liang, Yaming Yang, Shilin Xu, Tianmeng Yang, Yunhai Tong

专题命中 偏好对齐 ：alignment(abstract)；DPO(abstract)

Comments This paper has been accepted to ICME 2025

Advancing Tool-Augmented Large Language Models: Integrating Insights from Errors in Inference Trees

Sijia Chen, Yibo Wang, Yi-Feng Wu, Qing-Guo Chen, Zhao Xu, Weihua Luo, Kaifu Zhang, Lijun Zhang

专题命中 偏好对齐 ：DPO(abstract)；分类 cs.CL、cs.AI、cs.LG

Comments Accepted by NeurIPS 2024

Bayesian WeakS-to-Strong from Text Classification to Generation

Ziyun Cui, Ziyang Zhang, Guangzhi Sun, Wen Wu, Chao Zhang

专题命中 偏好对齐 ：alignment(abstract)；分类 cs.CL、cs.AI、cs.LG

Comments Accepted by ICLR2025

Rejected Dialects: Biases Against African American Language in Reward Models

Joel Mire, Zubin Trivadi Aysola, Daniel Chechelnitsky, Nicholas Deas, Chrysoula Zerva, Maarten Sap

专题命中 偏好对齐 ：alignment(abstract)；分类 cs.CL、cs.AI、cs.CY

Comments Accepted to NAACL Findings 2025

Enhancing Multi-Step Reasoning Abilities of Language Models through Direct Q-Function Optimization

Kaixuan Ji, Guanlin Liu, Ning Dai, Qingping Yang, Renjie Zheng, Zheng Wu, Chen Dun, Quanquan Gu, Lin Yan

专题命中 偏好对齐 ：DPO(abstract)；分类 cs.CL、cs.AI、cs.LG

Self-Generated Critiques Boost Reward Modeling for Language Models

Yue Yu, Zhengxing Chen, Aston Zhang, Liang Tan, Chenguang Zhu, Richard Yuanzhe Pang, Yundi Qian, Xuewei Wang, Suchin Gururangan, Chao Zhang, Melanie Kambadur, Dhruv Mahajan, Rui Hou

专题命中 偏好对齐 ：RLHF(abstract)；分类 cs.CL、cs.AI、cs.LG

Comments Accepted to NAACL 2025 (Main Conference)

Journal ref NAACL 2025

PerPO: Perceptual Preference Optimization via Discriminative Rewarding

Zining Zhu, Liang Zhao, Kangheng Lin, Jinze Yang, En Yu, Chenglong Liu, Haoran Wei, Jianjian Sun, Zheng Ge, Xiangyu Zhang

专题命中 偏好对齐 ：alignment(abstract)；分类 cs.CL、cs.AI、cs.LG

On Teacher Hacking in Language Model Distillation

Daniil Tiapkin, Daniele Calandriello, Johan Ferret, Sarah Perrin, Nino Vieillard, Alexandre Ramé, Mathieu Blondel

专题命中 偏好对齐 ：RLHF(abstract)；分类 cs.CL、cs.AI、cs.LG

Eliciting Language Model Behaviors with Investigator Agents

Xiang Lisa Li, Neil Chowdhury, Daniel D. Johnson, Tatsunori Hashimoto, Percy Liang, Sarah Schwettmann, Jacob Steinhardt

专题命中 偏好对齐 ：DPO(abstract)；分类 cs.CL、cs.AI、cs.LG

Comments 20 pages, 7 figures

Mitigating Hallucinated Translations in Large Language Models with Hallucination-focused Preference Optimization

Zilu Tang, Rajen Chatterjee, Sarthak Garg

专题命中 偏好对齐 ：safety(abstract)；分类 cs.CL、cs.AI、cs.LG

Comments NAACL 2025 Main Conference Long paper (9 pages)

Journal ref NAACL 2025

Aligning with Human Judgement: The Role of Pairwise Preference in Large Language Model Evaluators

Yinhong Liu, Han Zhou, Zhijiang Guo, Ehsan Shareghi, Ivan Vulić, Anna Korhonen, Nigel Collier

专题命中 偏好对齐 ：RLHF(abstract)；分类 cs.CL、cs.AI、cs.LG

Comments This paper has been accepted by COLM 2024

Critical Tokens Matter: Token-Level Contrastive Estimation Enhances LLM's Reasoning Capability

Zicheng Lin, Tian Liang, Jiahao Xu, Qiuzhi Lin, Xing Wang, Ruilin Luo, Chufan Shi, Siheng Li, Yujiu Yang, Zhaopeng Tu

专题命中 偏好对齐 ：DPO(abstract)；分类 cs.CL、cs.AI、cs.LG

Comments Work in progress

AGFSync: Leveraging AI-Generated Feedback for Preference Optimization in Text-to-Image Generation

Jingkun An, Yinghao Zhu, Zongjian Li, Enshen Zhou, Haoran Feng, Xijie Huang, Bohua Chen, Yemin Shi, Chengwei Pan

专题命中 偏好对齐 ：alignment(abstract)；DPO(abstract)

Comments Accepted by AAAI-2025

Offline Reinforcement Learning for LLM Multi-Step Reasoning

Huaijie Wang, Shibo Hao, Hanze Dong, Shenao Zhang, Yilin Bao, Ziran Yang, Yi Wu

专题命中 偏好对齐 ：DPO(abstract)；分类 cs.CL、cs.AI、cs.LG

A Post-Training Enhanced Optimization Approach for Small Language Models

Keke Zhai

专题命中 偏好对齐 ：alignment(abstract)；分类 cs.CL、cs.AI、cs.LG

Pipeline Analysis for Developing Instruct LLMs in Low-Resource Languages: A Case Study on Basque

Ander Corral, Ixak Sarasua, Xabier Saralegi

专题命中 偏好对齐 ：alignment(abstract)；分类 cs.CL、cs.AI、cs.LG

Why Does ChatGPT "Delve" So Much? Exploring the Sources of Lexical Overrepresentation in Large Language Models

Tom S. Juzek, Zina B. Ward

专题命中 偏好对齐 ：RLHF(abstract)；分类 cs.CL、cs.AI、cs.LG

Comments 15 pages, 8 figures, The 31st International Conference on Computational Linguistics (COLING 2025)

VickreyFeedback: Cost-efficient Data Construction for Reinforcement Learning from Human Feedback

Guoxi Zhang, Jiuding Duan

专题命中 偏好对齐 ：RLHF(abstract)；分类 cs.CL、cs.AI、cs.LG

Comments 16 pages, 5 figures

Imitating Language via Scalable Inverse Reinforcement Learning

Markus Wulfmeier, Michael Bloesch, Nino Vieillard, Arun Ahuja, Jorg Bornschein, Sandy Huang, Artem Sokolov, Matt Barnes, Guillaume Desjardins, Alex Bewley, Sarah Maria Elisabeth Bechtle, Jost Tobias Springenberg, Nikola Momchev, Olivier Bachem, Matthieu Geist, Martin Riedmiller

专题命中 偏好对齐 ：RLHF(abstract)；分类 cs.CL、cs.AI、cs.LG

Comments Published at NeurIPS 2024

Personalized Language Modeling from Personalized Human Feedback

Xinyu Li, Ruiyang Zhou, Zachary C. Lipton, Liu Leqi

专题命中 偏好对齐 ：RLHF(abstract)；分类 cs.CL、cs.AI、cs.LG

Self-Improvement in Language Models: The Sharpening Mechanism

Audrey Huang, Adam Block, Dylan J. Foster, Dhruv Rohatgi, Cyril Zhang, Max Simchowitz, Jordan T. Ash, Akshay Krishnamurthy

专题命中 偏好对齐 ：RLHF(abstract)；分类 cs.CL、cs.AI、cs.LG

Multimodal large language model for wheat breeding: a new exploration of smart breeding

Guofeng Yang, Yu Li, Yong He, Zhenjiang Zhou, Lingzhen Ye, Hui Fang, Yiqi Luo, Xuping Feng

专题命中 偏好对齐 ：RLHF(abstract)；分类 cs.CL、cs.AI、cs.LG

Aligning LLM Agents by Learning Latent Preference from User Edits

Ge Gao, Alexey Taymanov, Eduardo Salinas, Paul Mineiro, Dipendra Misra

专题命中 偏好对齐 ：alignment(abstract)；分类 cs.CL、cs.AI、cs.LG

Reward Modeling with Ordinal Feedback: Wisdom of the Crowd

Shang Liu, Yu Pan, Guanting Chen, Xiaocheng Li

专题命中 偏好对齐 ：DPO(abstract)；分类 cs.CL、cs.AI、cs.LG

Health Text Simplification: An Annotated Corpus for Digestive Cancer Education and Novel Strategies for Reinforcement Learning

Md Mushfiqur Rahman, Mohammad Sabik Irbaz, Kai North, Michelle S. Williams, Marcos Zampieri, Kevin Lybarger

专题命中 偏好对齐 ：RLHF(abstract)；分类 cs.CL、cs.AI、cs.LG

Comments Published in Journal of Biomedical Informatics, Volume 158, October 2024, 104727

MAVIS: Mathematical Visual Instruction Tuning with an Automatic Data Engine

Renrui Zhang, Xinyu Wei, Dongzhi Jiang, Ziyu Guo, Shicheng Li, Yichi Zhang, Chengzhuo Tong, Jiaming Liu, Aojun Zhou, Bin Wei, Shanghang Zhang, Peng Gao, Chunyuan Li, Hongsheng Li

专题命中 偏好对齐 ：alignment(abstract)；DPO(abstract)

Comments WData and Models will be released at https://github.com/ZrrSkywalker/MAVIS