大模型对齐与安全

Supervised Fine-Tuning as Inverse Reinforcement Learning

Hao Sun

专题命中 偏好对齐 ：alignment(abstract)；分类 cs.CL、cs.AI、cs.LG

$\mathbf{(N,K)}$-Puzzle: A Cost-Efficient Testbed for Benchmarking Reinforcement Learning Algorithms in Generative Language Model

Yufeng Zhang, Liyu Chen, Boyi Liu, Yingxiang Yang, Qiwen Cui, Yunzhe Tao, Hongxia Yang

专题命中 偏好对齐 ：DPO(abstract)；分类 cs.CL、cs.AI、cs.LG

Comments 8 pages

MINT: Evaluating LLMs in Multi-turn Interaction with Tools and Language Feedback

Xingyao Wang, Zihan Wang, Jiateng Liu, Yangyi Chen, Lifan Yuan, Hao Peng, Heng Ji

专题命中 偏好对齐 ：RLHF(abstract)；分类 cs.CL、cs.AI、cs.LG

Comments ICLR 2024. Code is available on our project website: https://xingyaoww.github.io/mint-bench

SELMA: Learning and Merging Skill-Specific Text-to-Image Experts with Auto-Generated Data

Jialu Li, Jaemin Cho, Yi-Lin Sung, Jaehong Yoon, Mohit Bansal

专题命中 偏好对齐 ：alignment(abstract)；分类 cs.CL、cs.AI、cs.LG

Comments First two authors contributed equally; Project website: https://selma-t2i.github.io/

CogBench: a large language model walks into a psychology lab

Julian Coda-Forno, Marcel Binz, Jane X. Wang, Eric Schulz

专题命中 偏好对齐 ：RLHF(abstract)；分类 cs.CL、cs.AI、cs.LG

RLVF: Learning from Verbal Feedback without Overgeneralization

Moritz Stephan, Alexander Khazatsky, Eric Mitchell, Annie S Chen, Sheryl Hsu, Archit Sharma, Chelsea Finn

专题命中 偏好对齐 ：RLHF(abstract)；分类 cs.CL、cs.AI、cs.LG

Comments 9 pages, 9 figures

DeepSeek LLM: Scaling Open-Source Language Models with Longtermism

DeepSeek-AI, :, Xiao Bi, Deli Chen, Guanting Chen, Shanhuang Chen, Damai Dai, Chengqi Deng, Honghui Ding, Kai Dong, Qiushi Du, Zhe Fu, Huazuo Gao, Kaige Gao, Wenjun Gao, Ruiqi Ge, Kang Guan, Daya Guo, Jianzhong Guo, Guangbo Hao, Zhewen Hao, Ying He, Wenjie Hu, Panpan Huang, Erhang Li, Guowei Li, Jiashi Li, Yao Li, Y. K. Li, Wenfeng Liang, Fangyun Lin, A. X. Liu, Bo Liu, Wen Liu, Xiaodong Liu, Xin Liu, Yiyuan Liu, Haoyu Lu, Shanghao Lu, Fuli Luo, Shirong Ma, Xiaotao Nie, Tian Pei, Yishi Piao, Junjie Qiu, Hui Qu, Tongzheng Ren, Zehui Ren, Chong Ruan, Zhangli Sha, Zhihong Shao, Junxiao Song, Xuecheng Su, Jingxiang Sun, Yaofeng Sun, Minghui Tang, Bingxuan Wang, Peiyi Wang, Shiyu Wang, Yaohui Wang, Yongji Wang, Tong Wu, Y. Wu, Xin Xie, Zhenda Xie, Ziwei Xie, Yiliang Xiong, Hanwei Xu, R. X. Xu, Yanhong Xu, Dejian Yang, Yuxiang You, Shuiping Yu, Xingkai Yu, B. Zhang, Haowei Zhang, Lecong Zhang, Liyue Zhang, Mingchuan Zhang, Minghua Zhang, Wentao Zhang, Yichao Zhang, Chenggang Zhao, Yao Zhao, Shangyan Zhou, Shunfeng Zhou, Qihao Zhu, Yuheng Zou

专题命中 偏好对齐 ：DPO(abstract)；分类 cs.CL、cs.AI、cs.LG

AI-TA: Towards an Intelligent Question-Answer Teaching Assistant using Open-Source LLMs

Yann Hicke, Anmol Agarwal, Qianou Ma, Paul Denny

专题命中 偏好对齐 ：DPO(abstract)；分类 cs.CL、cs.AI、cs.LG

Comments Updates for camera-ready submission

Journal ref NeurIPS Workshop on Generative AI for Education (GAIED), 2023

Language Models can Solve Computer Tasks

Geunwoo Kim, Pierre Baldi, Stephen McAleer

专题命中 偏好对齐 ：RLHF(abstract)；分类 cs.CL、cs.AI、cs.LG

Zero-Shot Goal-Directed Dialogue via RL on Imagined Conversations

Joey Hong, Sergey Levine, Anca Dragan

专题命中 偏好对齐 ：RLHF(abstract)；分类 cs.CL、cs.AI、cs.LG

Comments 25 pages, 6 figures

Reward Finetuning for Faster and More Accurate Unsupervised Object Discovery

Katie Z Luo, Zhenzhen Liu, Xiangyu Chen, Yurong You, Sagie Benaim, Cheng Perng Phoo, Mark Campbell, Wen Sun, Bharath Hariharan, Kilian Q. Weinberger

专题命中 偏好对齐 ：alignment(abstract)；RLHF(abstract)

Aligning Large Language Models through Synthetic Feedback

Sungdong Kim, Sanghwan Bae, Jamin Shin, Soyoung Kang, Donghyun Kwak, Kang Min Yoo, Minjoon Seo

专题命中 偏好对齐 ：alignment(abstract)；分类 cs.CL、cs.AI、cs.LG

Comments Accepted to EMNLP 2023 main conference

KwaiYiiMath: Technical Report

Jiayi Fu, Lei Lin, Xiaoyang Gao, Pengli Liu, Zhengzong Chen, Zhirui Yang, Shengnan Zhang, Xue Zheng, Yan Li, Yuliang Liu, Xucheng Ye, Yiqiao Liao, Chao Liao, Bin Chen, Chengru Song, Junchen Wan, Zijia Lin, Fuzheng Zhang, Zhongyuan Wang, Di Zhang, Kun Gai

专题命中 偏好对齐 ：RLHF(abstract)；分类 cs.CL、cs.AI、cs.LG

Comments technical report. arXiv admin note: text overlap with arXiv:2306.16636 by other authors

Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedback

Stephen Casper, Xander Davies, Claudia Shi, Thomas Krendl Gilbert, Jérémy Scheurer, Javier Rando, Rachel Freedman, Tomasz Korbak, David Lindner, Pedro Freire, Tony Wang, Samuel Marks, Charbel-Raphaël Segerie, Micah Carroll, Andi Peng, Phillip Christoffersen, Mehul Damani, Stewart Slocum, Usman Anwar, Anand Siththaranjan, Max Nadeau, Eric J. Michaud, Jacob Pfau, Dmitrii Krasheninnikov, Xin Chen, Lauro Langosco, Peter Hase, Erdem Bıyık, Anca Dragan, David Krueger, Dorsa Sadigh, Dylan Hadfield-Menell

专题命中 偏好对齐 ：RLHF(abstract)；分类 cs.CL、cs.AI、cs.LG

Do Models Explain Themselves? Counterfactual Simulatability of Natural Language Explanations

Yanda Chen, Ruiqi Zhong, Narutatsu Ri, Chen Zhao, He He, Jacob Steinhardt, Zhou Yu, Kathleen McKeown

专题命中 偏好对齐 ：RLHF(abstract)；分类 cs.CL、cs.AI、cs.LG

ILLUME: Rationalizing Vision-Language Models through Human Interactions

Manuel Brack, Patrick Schramowski, Björn Deiseroth, Kristian Kersting

专题命中 偏好对齐 ：alignment(abstract)；分类 cs.CL、cs.AI、cs.LG

Comments Proceedings of the 40th International Conference on Machine Learning (ICML), 2023

Chain-of-Thought Hub: A Continuous Effort to Measure Large Language Models' Reasoning Performance

Yao Fu, Litu Ou, Mingyu Chen, Yuhao Wan, Hao Peng, Tushar Khot

专题命中 偏好对齐 ：RLHF(abstract)；分类 cs.CL、cs.AI、cs.LG

Comments Preprint. Code at https://github.com/FranxYao/chain-of-thought-hub

DERA: Enhancing Large Language Model Completions with Dialog-Enabled Resolving Agents

Varun Nair, Elliot Schumacher, Geoffrey Tso, Anitha Kannan

专题命中 偏好对齐 ：safety(abstract)；分类 cs.CL、cs.AI、cs.LG

Discovering Language Model Behaviors with Model-Written Evaluations

Ethan Perez, Sam Ringer, Kamilė Lukošiūtė, Karina Nguyen, Edwin Chen, Scott Heiner, Craig Pettit, Catherine Olsson, Sandipan Kundu, Saurav Kadavath, Andy Jones, Anna Chen, Ben Mann, Brian Israel, Bryan Seethor, Cameron McKinnon, Christopher Olah, Da Yan, Daniela Amodei, Dario Amodei, Dawn Drain, Dustin Li, Eli Tran-Johnson, Guro Khundadze, Jackson Kernion, James Landis, Jamie Kerr, Jared Mueller, Jeeyoon Hyun, Joshua Landau, Kamal Ndousse, Landon Goldberg, Liane Lovitt, Martin Lucas, Michael Sellitto, Miranda Zhang, Neerav Kingsland, Nelson Elhage, Nicholas Joseph, Noemí Mercado, Nova DasSarma, Oliver Rausch, Robin Larson, Sam McCandlish, Scott Johnston, Shauna Kravec, Sheer El Showk, Tamera Lanham, Timothy Telleen-Lawton, Tom Brown, Tom Henighan, Tristan Hume, Yuntao Bai, Zac Hatfield-Dodds, Jack Clark, Samuel R. Bowman, Amanda Askell, Roger Grosse, Danny Hernandez, Deep Ganguli, Evan Hubinger, Nicholas Schiefer, Jared Kaplan

专题命中 偏好对齐 ：RLHF(abstract)；分类 cs.CL、cs.AI、cs.LG

Comments for associated data visualizations, see https://www.evals.anthropic.com/model-written/ for full datasets, see https://github.com/anthropics/evals

Learning Norms from Stories: A Prior for Value Aligned Agents

Spencer Frazier, Md Sultan Al Nahian, Mark Riedl, Brent Harrison

专题命中 偏好对齐 ：alignment(abstract)；分类 cs.CL、cs.AI、cs.LG

Comments AIES 2020

Steerable Cultural Preference Optimization of Reward Models

可引导的文化偏好优化奖励模型

Minsik Oh, Advit Deepak, Sophie Wu, Douwe Kiela, Ekaterina Shutova

机构 * Stanford University（斯坦福大学） ； University of Amsterdam（阿姆斯特丹大学） 纠错

专题命中 偏好对齐 ：alignment(abstract,comments)；分类 cs.CL、cs.AI

AI总结 提出SCPO算法，通过平衡多种文化偏好训练奖励模型，在PRISM和GlobalOpinionQA数据集上提升少数群体偏好预测准确率最多7点，训练效率提高280%。

Comments Accepted to Pluralistic Alignment @ ICML 2026

Moral Change or Noise? On Problems of Aligning AI With Temporally Unstable Human Feedback

Vijay Keswani, Cyrus Cousins, Breanna Nguyen, Vincent Conitzer, Hoda Heidari, Jana Schaich Borg, Walter Sinnott-Armstrong

机构 * Duke University（杜克大学） ； Carnegie Mellon University（卡内基梅隆大学） 纠错

专题命中 偏好对齐 ：alignment(abstract,comments)；分类 cs.AI、cs.CY

Comments To appear in the AAAI 2026 Alignment Track

Understanding Impact of Human Feedback via Influence Functions

Taywon Min, Haeone Lee, Yongchan Kwon, Kimin Lee

机构 * KAIST（韩国科学技术院） ； Columbia University（哥伦比亚大学） 纠错

专题命中 偏好对齐 ：RLHF(abstract,comments)；分类 cs.AI、cs.LG

Comments Accepted at ACL 2025, Source code: https://github.com/mintaywon/IF_RLHF

Journal ref Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics 63 (2025) 27471-27500

Multi-Task Reward Learning from Human Ratings

Mingkang Wu, Devin White, Evelyn Rose, Vernon Lawhern, Nicholas R Waytowich, Yongcan Cao

机构 * Department of Electrical and Computer Engineering, University of Texas at San Antonio, Texas, USA（电子与计算机工程系，德克萨斯州立大学圣安东尼奥分校） ； DEVCOM Army Research Lab, USA（陆军研究实验室） 纠错

专题命中 偏好对齐 ：RLHF(abstract)；分类 cs.AI、cs.LG；alignment(comments)

Comments Accepted to the workshop on Models of Human Feedback for AI Alignment at the 42nd International Conference on Machine Learning

Reliability Check: An Analysis of GPT-3's Response to Sensitive Topics and Prompt Wording

Aisha Khatun, Daniel G. Brown

专题命中 偏好对齐 ：RLHF(abstract)；分类 cs.CL、cs.LG；trustworthy(comments)

Comments Accepted in TrustNLP: Third Workshop on Trustworthy Natural Language Processing, co-located with ACL 2023

EstLLM: Enhancing Estonian Capabilities in Multilingual LLMs via Continued Pretraining and Post-Training

EstLLM：通过持续预训练和后训练增强多语言大语言模型中的爱沙尼亚能力

Aleksei Dorkin, Taido Purason, Emil Kalbaliyev, Hele-Andra Kuulmets, Marii Ojastu, Mark Fišel, Tanel Alumäe, Eleri Aedmaa, Krister Kruusmaa, Kairit Sirts

机构 * Institute of Computer Science, University of Tartu（塔尔图大学计算机科学研究院） ； Department of Software Science, Tallinn University of Technology（塔林技术大学软件科学系） ； Institute of the Estonian Language, Tallinn, Estonia（爱沙尼亚语言研究院） ； School of Humanities, Tallinn University（塔林大学人文学院） 纠错

专题命中 偏好对齐 ：alignment(abstract)；分类 cs.CL、cs.AI

AI总结 EstLLM通过持续预训练和后训练提升多语言大语言模型中爱沙尼亚的能力，增强语言和推理表现。

Preference Reasoning under Indeterminacy in Large Language Models

大语言模型在不确定性下的偏好推理

Hadi Hosseini, Samarth Khanna, Xiyuan Wang

机构 * Penn State University（宾夕法尼亚州立大学） 纠错

专题命中 偏好对齐 ：alignment(abstract)；分类 cs.AI、cs.LG

AI总结 本文聚焦大语言模型决策智能体的偏好推理问题，将不确定性挑战形式化为认知与结构两类，发现当前模型无法区分确定与不确定实例，推理校准不当。

Comments 55 pages, 14 figures

Whose Gold? Annotator-Pool Disagreement Is Large at the Item Level, and Hidden by Small Leaderboards

谁的“金标准”？标注者群体在条目层面分歧巨大，却被小型排行榜掩盖

Anik Jha

专题命中 偏好对齐 ：safety(abstract)；分类 cs.CL、cs.LG

AI总结 该研究发现标注者群体在条目层面分歧巨大，小型模型排行榜的一致性是假象，量化了其脆弱性，证明某数据集的标注者无差异假设错误，LLM评判器更贴合大众标注者群体。

Comments Submitted to the HAIC workshop at NeurIPS 2026

PEER: Unified Process-Outcome Reinforcement Learning for Structured Empathetic Reasoning

PEER：统一的过程-结果强化学习用于结构化共情推理

Yunxiao Wang, Meng Liu, Sicheng Zhao, Lizi Liao, Liqiang Nie

机构 * Shandong University（山东大学） ； Shandong Jianzhu University（山东建筑大学） ； Singapore Management University（新加坡管理大学） 纠错

专题命中 偏好对齐 ：alignment(abstract)；分类 cs.CL、cs.AI

AI总结 本文提出PEER，通过结构化共情推理框架提升情感支持对话的 empathy、策略一致性及人性表现，采用GRPO与UnifiReward模型，结合数据增强减少重复。

VDC-Agent: When Video Detailed Captioners Evolve Themselves via Agentic Self-Reflection

VDC-Agent：当视频详细描述器通过代理自我反思而自我进化

Qiang Wang, Xinyuan Gao, Yuhang He, Jizhou Han, Jiangyang Li, SongLin Dong, Zhiheng Ma, Yihong Gong

机构 * Xi’an Jiaotong University（西安交通大学） ； Kuaishou Technology（快手科技） ； Shenzhen University of Advanced Technology（深圳先进技术大学） 纠错

专题命中 偏好对齐 ：DPO(abstract)；分类 cs.AI、cs.LG

AI总结 VDC-Agent通过自我反思机制实现视频详细描述的自我进化，利用自动生成的(描述，评分)对提升描述准确性与评分表现。

Comments Accepted to ECCV 2026. Project Page: https://vdcagent.github.io