大模型对齐与安全 - arXivDaily 专题

2406.00037 2024-06-04 cs.CL cs.AI 73%

Aligning LLMs through Multi-perspective User Preference Ranking-based Feedback for Programming Question Answering

Hongyu Yang, Liyang He, Min Hou, Shuanghong Shen, Rui Li, Jiahui Hou, Jianhui Ma, Junda Zhao

专题命中偏好对齐：alignment(abstract)；RLHF(abstract)；分类 cs.CL、cs.AI

2405.20175 2024-05-31 cs.CL cs.AI 73%

InstructionCP: A fast approach to transfer Large Language Models into target language

Kuang-Ming Chen, Hung-yi Lee

专题命中偏好对齐：alignment(abstract)；RLHF(abstract)；分类 cs.CL、cs.AI

Comments 10 pages, 1 figure

2310.12931 2024-05-02 cs.RO cs.AI cs.LG 73%

Eureka: Human-Level Reward Design via Coding Large Language Models

Yecheng Jason Ma, William Liang, Guanzhi Wang, De-An Huang, Osbert Bastani, Dinesh Jayaraman, Yuke Zhu, Linxi Fan, Anima Anandkumar

专题命中偏好对齐：RLHF(abstract)；safety(abstract)；分类 cs.AI、cs.LG

Comments ICLR 2024. Project website and open-source code: https://eureka-research.github.io/

2404.13076 2024-04-23 cs.CL cs.AI 73%

LLM Evaluators Recognize and Favor Their Own Generations

Arjun Panickssery, Samuel R. Bowman, Shi Feng

专题命中偏好对齐：safety(abstract)；AI safety(abstract)；分类 cs.CL、cs.AI

2404.12195 2024-04-19 cs.CL cs.LG 73%

OpenBezoar: Small, Cost-Effective and Open Models Trained on Mixes of Instruction Data

Chandeepa Dissanayake, Lahiru Lowe, Sachith Gunasekara, Yasiru Ratnayake

专题命中偏好对齐：RLHF(abstract)；DPO(abstract)；分类 cs.CL、cs.LG

Comments 25 pages, 27 Figures, 8 Tables

2404.12150 2024-04-19 cs.LG cs.CL 73%

Aligning language models with human preferences

Tomasz Korbak

专题命中偏好对齐：alignment(abstract)；RLHF(abstract)；分类 cs.CL、cs.LG

Comments PhD thesis

2404.07439 2024-04-12 cs.AI cs.CL 73%

Behavior Trees Enable Structured Programming of Language Model Agents

Richard Kelley

专题命中偏好对齐：RLHF(abstract)；safety(abstract)；分类 cs.CL、cs.AI

2312.16430 2024-03-26 cs.LG cs.AI 73%

Preference as Reward, Maximum Preference Optimization with Importance Sampling

Zaifan Jiang, Xing Huang, Chao Wei

专题命中偏好对齐：RLHF(abstract)；DPO(abstract)；分类 cs.AI、cs.LG

2311.13231 2024-03-26 cs.LG cs.AI cs.CV 73%

Using Human Feedback to Fine-tune Diffusion Models without Any Reward Model

Kai Yang, Jian Tao, Jiafei Lyu, Chunjiang Ge, Jiaxin Chen, Qimai Li, Weihan Shen, Xiaolong Zhu, Xiu Li

专题命中偏好对齐：RLHF(abstract)；DPO(abstract)；分类 cs.AI、cs.LG

Comments CVPR 2024 accepted; huggingface daily paper

2403.14238 2024-03-22 cs.CL cs.AI 73%

Reinforcement Learning from Reflective Feedback (RLRF): Aligning and Improving LLMs via Fine-Grained Self-Reflection

Kyungjae Lee, Dasol Hwang, Sunghyun Park, Youngsoo Jang, Moontae Lee

专题命中偏好对齐：alignment(abstract)；RLHF(abstract)；分类 cs.CL、cs.AI

Comments 22 pages, 5 figures, Submitted to ACL 2024

2402.11411 2024-02-20 cs.LG cs.CL cs.CV 73%

Aligning Modalities in Vision Large Language Models via Preference Fine-tuning

Yiyang Zhou, Chenhang Cui, Rafael Rafailov, Chelsea Finn, Huaxiu Yao

专题命中偏好对齐：alignment(abstract)；RLHF(abstract)；分类 cs.CL、cs.LG

2311.14743 2024-01-25 cs.CL cs.LG 73%

A Baseline Analysis of Reward Models' Ability To Accurately Analyze Foundation Models Under Distribution Shift

Will LeVine, Benjamin Pikus, Anthony Chen, Sean Hendryx

专题命中偏好对齐：alignment(abstract)；RLHF(abstract)；分类 cs.CL、cs.LG

2310.10076 2023-10-17 cs.CL cs.AI 73%

Verbosity Bias in Preference Labeling by Large Language Models

Keita Saito, Akifumi Wachi, Koki Wataoka, Youhei Akimoto

专题命中偏好对齐：alignment(abstract)；RLHF(abstract)；分类 cs.CL、cs.AI

2310.02527 2023-10-05 cs.CL cs.AI 73%

CITING: Large Language Models Create Curriculum for Instruction Tuning

Tao Feng, Zifeng Wang, Jimeng Sun

专题命中偏好对齐：alignment(abstract)；RLHF(abstract)；分类 cs.CL、cs.AI

2302.08215 2023-06-07 cs.CL cs.LG stat.ML 73%

Aligning Language Models with Preferences through f-divergence Minimization

Dongyoung Go, Tomasz Korbak, Germán Kruszewski, Jos Rozen, Nahyeon Ryu, Marc Dymetman

专题命中偏好对齐：alignment(abstract)；RLHF(abstract)；分类 cs.CL、cs.LG

2302.05206 2023-02-13 cs.CL cs.AI 73%

The Wisdom of Hindsight Makes Language Models Better Instruction Followers

Tianjun Zhang, Fangchen Liu, Justin Wong, Pieter Abbeel, Joseph E. Gonzalez

专题命中偏好对齐：alignment(abstract)；RLHF(abstract)；分类 cs.CL、cs.AI

2409.13948 2025-02-25 cs.CL 72%

Aligning Language Models Using Follow-up Likelihood as Reward Signal

Chen Zhang, Dading Chong, Feng Jiang, Chengguang Tang, Anningzhe Gao, Guohua Tang, Haizhou Li

专题命中偏好对齐：alignment(abstract,comments)；DPO(abstract)；分类 cs.CL

Comments Accepted by AAAI-2025, 16 pages, reward model, LLM Alignment, code repository at (https://github.com/e0397123/FLR)

2603.19665 2026-08-24 cs.IR 71%

GenFacet: End-to-End Generative Faceted Search via Multi-Task Preference Alignment in E-Commerce

GenFacet：通过多任务偏好对齐的端到端生成方法实现电商多维搜索

Zhouwei Zhai, Min Yang, Jin Li

专题命中偏好对齐：alignment(title)

AI总结本文提出GenFacet，一种端到端生成框架，通过多任务偏好对齐提升电商多维搜索效果，实验证明显著提升点击率和转化率。

Journal ref Proceedings of the 49th International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR 2026), July 20-24, 2026, Melbourne, VIC, Australia

2512.19632 2025-12-23 cs.CV 71%

Generative diffusion models for agricultural AI: plant image generation, indoor-to-outdoor translation, and expert preference alignment

生成扩散模型在农业AI中的应用：植物图像生成、室内外转换以及专家偏好对齐

Da Tan, Michael Beck, Christopher P. Bidinosti, Robert H. Gulden, Christopher J. Henry

机构 * University of Manitoba（曼尼托巴大学）； University of Winnipeg（温哥华大学）

专题命中偏好对齐：alignment(title)

AI总结本文提出基于扩散模型的生成方法，通过合成植物图像、室内外转换和专家偏好对齐，提升农业AI的数据效率和性能。

2504.13444 2025-11-07 econ.GN q-fin.EC 71%

Balancing Engagement and Polarization: Multi-Objective Alignment of News Content Using LLMs

Mengjie Cheng, Elie Ofek, Hema Yoganarasimhan

专题命中偏好对齐：alignment(title)

Comments 73 pages

2509.12750 2025-09-17 cs.CV 71%

What Makes a Good Generated Image? Investigating Human and Multimodal LLM Image Preference Alignment

Rishab Parthasarathy, Jasmine Collins, Cory Stephenson

专题命中偏好对齐：alignment(title)

Comments 7 pages, 9 figures, 3 tables; appendix 16 pages, 9 figures, 6 tables

2508.06036 2025-08-11 cs.CV 71%

More Is Better: A MoE-Based Emotion Recognition Framework with Human Preference Alignment

Jun Xie, Yingjian Zhu, Feng Chen, Zhenghao Zhang, Xiaohui Fan, Hongzhu Yi, Xinming Wang, Chen Yu, Yue Bi, Zhaoran Zhao, Xiongjun Guan, Zhepeng Wang

机构 * Lenovo Research（联想研究）； Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）； University of Chinese Academy of Sciences（中国科学院大学）； Tsinghua University（清华大学）； Beijing Jiaotong University（北京交通大学）； Shandong University（山东大学）

专题命中偏好对齐：alignment(title)

2412.19326 2025-07-01 cs.CV 71%

Task Preference Optimization: Improving Multimodal Large Language Models with Vision Task Alignment

Ziang Yan, Zhilin Li, Yinan He, Chenting Wang, Kunchang Li, Xinhao Li, Xiangyu Zeng, Zilei Wang, Yali Wang, Yu Qiao, Limin Wang, Yi Wang

机构 * Shanghai AI Laboratory（上海人工智能实验室）； Zhejiang University（浙江大学）； University of Science and Technology of China（中国科学技术大学）； Shanghai Jiao Tong University（上海交通大学）； Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences（中国科学院深圳先进技术研究所）； Nanjing University（南京大学）； Shanghai Innovation Institute（上海创新研究院）

专题命中偏好对齐：alignment(title)

Comments CVPR2025

2608.25350 2026-08-27 cs.LG cs.RO 新提交 70%

Beyond Pairwise Feedback: Listwise Vision-Language Supervision for Preference-Based Reward Learning

超越成对反馈：用于基于偏好的奖励学习的列表式视觉-语言监督

Srivalli Katkuri, Maxwell Kawada, Juan Wachs

专题命中偏好对齐：RLHF(abstract,abstract_cn)；分类 cs.LG

AI总结本研究提出首个结合视觉-语言模型生成偏好与普拉科特-卢斯模型的列表式奖励学习框架，在Meta-World操纵任务中，其表现与基线相当且更灵活，最佳配置达86%平均成功率。

Comments 13 pages, 10 figures. Srivalli Katkuri and Maxwell Kawada contributed equally to this work

2604.07054 2026-08-27 cs.CL 版本更新 70%

Sell More, Play Less: Benchmarking LLM Realistic Selling Skill

多卖少玩：LLM真实销售技能基准测试

Xuanbo Su, Wenhao Hu, Le Zhan, Yuting Xie, Kailin Lyu, Kaijie Chen, Ziwei Li, Yeqiang Wang, Haibo Su, Yunzhang Chen, Ling Huang

机构 * SF Express（顺丰速运）

专题命中偏好对齐：DPO(abstract,abstract_cn)；分类 cs.CL

AI总结本文提出SalesLLM基准测试，通过30,074个剧本配置和1,805个多轮场景评估大语言模型的销售能力，采用自动评估流程和用户模型CustomerLM提升模拟真实性，实验显示LLM在销售任务上的表现差异显著。

2504.19342 2026-08-26 stat.ML cs.LG stat.ME 版本更新 70%

Contextual Online Uncertainty-Aware Preference Learning for Human Feedback

基于上下文的在线不确定性感知偏好学习用于人类反馈

Nan Lu, Ethan Lee, Ethan X. Fang, Junwei Lu

机构 * Department of Biostatistics, Harvard T.H. Chan School of Public Health（哈佛大学T.H.陈公共卫生学院生物统计学系）； Department of Biostatistics & Bioinformatics, Duke University（杜克大学生物统计学与生物信息学系）

专题命中偏好对齐：RLHF(abstract,abstract_cn)；分类 cs.LG

AI总结本文提出一种新的统计框架，利用动态上下文信息在线决策和统计推断最优模型，通过人类偏好数据实现最优 regret 绑定和估计量渐近分布。

2504.08846 2026-08-25 cs.CY cs.AI cs.CL cs.LG 版本更新 70%

AI University: An LLM-Powered Learning Assistant for Engineering---A Finite Element Method Case Study

AI University：一款面向工程领域的大语言模型驱动学习助手——以有限元法案例研究为例

Mostafa Faghih Shojaei, Rahul Gulati, Benjamin A. Jasperson, Shangshang Wang, Simone Cimolato, Manas Vardhan, Dangli Cao, Willie Neiswanger, Krishna Garikipati

专题命中偏好对齐：alignment(abstract)；分类 cs.CL、cs.AI、cs.CY

AI总结本研究提出AI-U框架，结合微调LLM、RAG与推理合成模型，以FEM课程为案例开发学习助手，经多维度评估其对齐度优于基础模型，可推广至STEM领域。

Comments 14 pages, 3 figures

2509.19893 2026-08-20 cs.CL 版本更新 70%

Future Policy Approximation for Offline Reinforcement Learning in LLM Reasoning

为离线强化学习未来政策近似改进数学推理

Minjae Oh, Yunho Choi, Dongmin Choi, Yohan Jo

机构 * Graduate School of Data Science, Seoul National University（首尔大学数据科学研究生院）

专题命中偏好对齐：DPO(abstract,abstract_cn)；分类 cs.CL

AI总结本文提出未来政策近似方法，通过估计未来策略来优化离线强化学习中的梯度更新，提升长周期推理任务的稳定性与准确性。

Comments 12 pages

2606.23671 2026-08-19 cs.CL 版本更新 70%

Can LLMs Reliably Self-Report Adversarial Prefills, and How?

LLM 能否可靠地自我报告对抗性预填充，以及如何实现？

Quang Minh Nguyen, Uzair Ahmed, Taegyoon Kim

机构 * KAIST（韩国科学技术院）

专题命中偏好对齐：DPO(abstract_cn)；safety(abstract)；分类 cs.CL

AI总结研究LLM在安全场景中识别自身输出是否受对抗性预填充攻击的能力，发现模型平均27.3%声称预填充输出有意图，且内省信号主要来自安全/拒绝推理，LoRA微调方法扩大了意图-篡改探针差距并提高了攻击成功率。

Comments In submission

2608.11152 2026-08-12 cs.DC cs.LG 新提交 70%

Scheduling Mixed RL Rollouts Beyond Prefix Locality

超越前缀局部性的混合强化学习回滚调度

Zetao Hong, Song Yuan, Yuanhao Ding, Yibo Zhu, Daxin Jiang, Zhibin Wang, Chen Tian

专题命中偏好对齐：RLHF(abstract,abstract_cn)；分类 cs.LG

AI总结针对混合RL回滚调度的异构性问题，提出MISA-T策略，在多基准实验中显著提升回滚吞吐量并降低平均轮次时间，同时维持缓存命中率与工作负载混合比例。

1. 偏好对齐 3273 篇

Aligning LLMs through Multi-perspective User Preference Ranking-based Feedback for Programming Question Answering

InstructionCP: A fast approach to transfer Large Language Models into target language

Eureka: Human-Level Reward Design via Coding Large Language Models

LLM Evaluators Recognize and Favor Their Own Generations

OpenBezoar: Small, Cost-Effective and Open Models Trained on Mixes of Instruction Data

Aligning language models with human preferences

Behavior Trees Enable Structured Programming of Language Model Agents

Preference as Reward, Maximum Preference Optimization with Importance Sampling

Using Human Feedback to Fine-tune Diffusion Models without Any Reward Model

Reinforcement Learning from Reflective Feedback (RLRF): Aligning and Improving LLMs via Fine-Grained Self-Reflection

Aligning Modalities in Vision Large Language Models via Preference Fine-tuning

A Baseline Analysis of Reward Models' Ability To Accurately Analyze Foundation Models Under Distribution Shift

Verbosity Bias in Preference Labeling by Large Language Models

CITING: Large Language Models Create Curriculum for Instruction Tuning

Aligning Language Models with Preferences through f-divergence Minimization

The Wisdom of Hindsight Makes Language Models Better Instruction Followers

Aligning Language Models Using Follow-up Likelihood as Reward Signal

GenFacet: End-to-End Generative Faceted Search via Multi-Task Preference Alignment in E-Commerce

Generative diffusion models for agricultural AI: plant image generation, indoor-to-outdoor translation, and expert preference alignment

Balancing Engagement and Polarization: Multi-Objective Alignment of News Content Using LLMs

What Makes a Good Generated Image? Investigating Human and Multimodal LLM Image Preference Alignment

More Is Better: A MoE-Based Emotion Recognition Framework with Human Preference Alignment

Task Preference Optimization: Improving Multimodal Large Language Models with Vision Task Alignment

Beyond Pairwise Feedback: Listwise Vision-Language Supervision for Preference-Based Reward Learning

Sell More, Play Less: Benchmarking LLM Realistic Selling Skill

Contextual Online Uncertainty-Aware Preference Learning for Human Feedback

AI University: An LLM-Powered Learning Assistant for Engineering---A Finite Element Method Case Study

Future Policy Approximation for Offline Reinforcement Learning in LLM Reasoning

Can LLMs Reliably Self-Report Adversarial Prefills, and How?

Scheduling Mixed RL Rollouts Beyond Prefix Locality