大模型对齐与安全 - arXivDaily 专题

2512.00042 2025-12-02 cs.CV cs.AI cs.CL cs.CY 67%

Closing the Gap: Data-Centric Fine-Tuning of Vision Language Models for the Standardized Exam Questions

弥合差距：面向标准化考试题目的视觉语言模型数据驱动微调

Egemen Sert, Şeyda Ertekin

机构 * organization= Department of Computer Engineering, Middle East Technical University (METU) , city= Ankara , country= Türkiye ； organization= METU-DTX Digital Transformation \& Innovation Centre, METU , city= Ankara , country= Türkiye

专题命中偏好对齐：DPO(abstract)；分类 cs.CL、cs.AI、cs.CY

AI总结本研究通过高质量数据和优化语法提升视觉语言模型在标准化考试题目的多模态推理性能，达到接近SOTA的水平。

2511.19523 2025-11-26 cs.CR 67%

EAGER: Edge-Aligned LLM Defense for Robust, Efficient, and Accurate Cybersecurity Question Answering

EAGER: 边缘对齐的LLM防御方法用于鲁棒、高效且准确的网络安全问答

Onat Gungor, Roshan Sood, Jiasheng Zhou, Tajana Rosing

专题命中偏好对齐：alignment(abstract)；DPO(abstract)

AI总结 EAGER通过参数高效量化与领域偏好对齐，提升网络安全问答的鲁棒性、效率和准确性，降低对抗攻击成功率并优化响应延迟。

2511.12036 2025-11-18 cs.CE cond-mat.mtrl-sci cs.AI cs.CL cs.LG 67%

Preference Learning from Physics-Based Feedback: Tuning Language Models to Design BCC/B2 Superalloys

Satanu Ghosh, Collin Holgate, Neal R. Brodnik, Doug Downey, Samantha Daly, Tresa M. Pollock, Samuel Carton

机构 * Department of Computer Science, University of New Hampshire（新罕布什尔大学计算机科学系）； Materials Department, University of California, Santa Barbara（加州大学圣芭芭拉分校材料系）； Department of Mechanical Engineering, University of California, Santa Barbara（加州大学圣芭芭拉分校机械工程系）； Allen Institute for Artificial Intelligence（人工智能研究院）

专题命中偏好对齐：DPO(abstract)；分类 cs.CL、cs.AI、cs.LG

2410.21597 2025-11-14 cs.CL cs.AI cs.LG 67%

Reducing the Scope of Language Models

David Yunis, Siyu Huo, Chulaka Gunasekara, Danish Contractor

机构 * Toyota Technological Institute at Chicago（丰田技术研究所（芝加哥））； IBM（IBM公司）

专题命中偏好对齐：alignment(abstract)；分类 cs.CL、cs.AI、cs.LG

Comments Appears in AAAI 2026 in the Main Technical Track

2511.08394 2025-11-12 cs.CL cs.AI cs.HC cs.LG 67%

Interaction Dynamics as a Reward Signal for LLMs

Sian Gooding, Edward Grefenstette

机构 * Google DeepMind（谷歌DeepMind）

专题命中偏好对齐：alignment(abstract)；分类 cs.CL、cs.AI、cs.LG

2510.25884 2025-10-31 cs.AI cs.CL cs.LG 67%

Approximating Human Preferences Using a Multi-Judge Learned System

Eitán Sprejer, Fernando Avalos, Augusto Bernardi, Jose Pedro Brito de Azevedo Faustino, Jacob Haimes, Narmeen Fatimah Oozeer

专题命中偏好对齐：RLHF(abstract)；分类 cs.CL、cs.AI、cs.LG

2505.02820 2025-10-31 cs.AI cs.CL cs.LG 67%

AutoLibra: Agent Metric Induction from Open-Ended Human Feedback

Hao Zhu, Phil Cuvin, Xinkai Yu, Charlotte Ka Yee Yan, Jason Zhang, Diyi Yang

机构 * Stanford University（斯坦福大学）； University of Toronto（多伦多大学）； University of Pennsylvania（宾夕法尼亚大学）

专题命中偏好对齐：alignment(abstract)；分类 cs.CL、cs.AI、cs.LG

Comments https://github.com/Open-Social-World/autolibra

2505.19601 2025-10-28 cs.LG cs.AI cs.CL 67%

Preference Optimization by Estimating the Ratio of the Data Distribution

Yeongmin Kim, Heesun Bae, Byeonghu Na, Il-Chul Moon

机构 * Korea Advanced Institute of Science and Technology (KAIST)（韩国科学技术院）

专题命中偏好对齐：DPO(abstract)；分类 cs.CL、cs.AI、cs.LG

Comments NeurIPS 2025

2504.10637 2025-10-28 cs.CL cs.AI cs.LG 67%

Better Estimation of the Kullback--Leibler Divergence Between Language Models

Afra Amini, Tim Vieira, Ryan Cotterell

机构 * ETH Zürich（苏黎世联邦理工学院）

专题命中偏好对齐：RLHF(abstract)；分类 cs.CL、cs.AI、cs.LG

Comments NeurIPS 2025

2505.11475 2025-10-27 cs.CL cs.AI cs.LG 67%

HelpSteer3-Preference: Open Human-Annotated Preference Data across Diverse Tasks and Languages

Zhilin Wang, Jiaqi Zeng, Olivier Delalleau, Hoo-Chang Shin, Felipe Soares, Alexander Bukharin, Ellie Evans, Yi Dong, Oleksii Kuchaiev

机构 * NVIDIA

专题命中偏好对齐：RLHF(abstract)；分类 cs.CL、cs.AI、cs.LG

Comments NeurIPS 2025 Datasets and Benchmarks Track Camera Ready, 46 pages, 2 figures

2505.11080 2025-10-27 cs.CL cs.AI cs.LG 67%

BLEUBERI: BLEU is a surprisingly effective reward for instruction following

Yapei Chang, Yekyung Kim, Michael Krumdick, Amir Zadeh, Chuan Li, Chris Tanner, Mohit Iyyer

机构 * University of Maryland, College Park（马里兰大学学院公园分校）

专题命中偏好对齐：alignment(abstract)；分类 cs.CL、cs.AI、cs.LG

Comments neurips cam-ready

2509.17999 2025-10-22 cs.CY cs.AI cs.HC cs.LG 67%

The Narcissus Hypothesis: Descending to the Rung of Illusion

Riccardo Cadei, Christian Internò

专题命中偏好对齐：alignment(abstract)；分类 cs.AI、cs.CY、cs.LG

Comments NeurIPS 2025 Workshop on Evaluating the Evolving LLM Lifecycle: Benchmarks, Emergent Abilities, and Scaling

2506.03517 2025-10-13 cs.CV 67%

DenseDPO: Fine-Grained Temporal Preference Optimization for Video Diffusion Models

Ziyi Wu, Anil Kag, Ivan Skorokhodov, Willi Menapace, Ashkan Mirzaei, Igor Gilitschenski, Sergey Tulyakov, Aliaksandr Siarohin

机构 * Snap Research ； University of Toronto（多伦多大学）； Vector Institute（向量研究所）

专题命中偏好对齐：alignment(abstract)；DPO(abstract)

Comments NeurIPS 2025 Spotlight. Project page: https://snap-research.github.io/DenseDPO/

2510.07557 2025-10-10 cs.LG cs.AI cs.CY cs.HC 67%

Investigating Thematic Patterns and User Preferences in LLM Interactions using BERTopic

Abhay Bhandarkar, Gaurav Mishra, Khushi Juchani, Harsh Singhal

机构 * Ramaiah Institute of Technology（拉马亚院技术学院）； Ecofy Finance Private Limited（Ecofy金融私人有限公司）

专题命中偏好对齐：alignment(abstract)；分类 cs.AI、cs.CY、cs.LG

2506.08022 2025-10-10 cs.LG cs.AI cs.CL cs.CV 67%

Modality-Balancing Preference Optimization of Large Multimodal Models by Adversarial Negative Mining

Chenxi Liu, Tianyi Xiong, Yanshuo Chen, Ruibo Chen, Yihan Wu, Junfeng Guo, Tianyi Zhou, Heng Huang

机构 * University of Maryland, College Park（马里兰大学学院 park）

专题命中偏好对齐：alignment(abstract)；分类 cs.CL、cs.AI、cs.LG

2503.03654 2025-10-09 cs.CL cs.AI cs.LG 67%

Improving Neutral Point-of-View Generation with Data- and Parameter-Efficient RL

Jessica Hoffmann, Christiane Ahlheim, Zac Yu, Aria Walfrand, Jarvis Jin, Marie Tano, Ahmad Beirami, Erin van Liemt, Nithum Thain, Hakim Sidahmed, Lucas Dixon

机构 * Google DeepMind（谷歌DeepMind）

专题命中偏好对齐：RLHF(abstract)；分类 cs.CL、cs.AI、cs.LG

2502.03323 2025-10-03 cs.CL cs.AI cs.LG 67%

Out-of-Distribution Detection using Synthetic Data Generation

Momin Abbas, Muneeza Azmat, Raya Horesh, Mikhail Yurochkin

专题命中偏好对齐：RLHF(abstract)；分类 cs.CL、cs.AI、cs.LG

Comments Accepted to COLM 2025. Camera-ready version

2509.26231 2025-10-01 cs.CV 67%

IMG: Calibrating Diffusion Models via Implicit Multimodal Guidance

Jiayi Guo, Chuanhao Yan, Xingqian Xu, Yulin Wang, Kai Wang, Gao Huang, Humphrey Shi

机构 * Tsinghua University（清华大学）

专题命中偏好对齐：alignment(abstract)；DPO(abstract)

Comments ICCV 2025

2509.25409 2025-10-01 cs.CL cs.AI cs.LG 67%

From Faithfulness to Correctness: Generative Reward Models that Think Critically

Qiyao Ma, Yunsheng Shi, Hongtao Tian, Chao Wang, Weiming Chang, Ting Yao

机构 * University of California, Davis（加州大学戴维斯分校）； WeChat, Tencent（微信、腾讯）； Tsinghua University（清华大学）

专题命中偏好对齐：alignment(abstract)；分类 cs.CL、cs.AI、cs.LG

2506.03006 2025-10-01 cs.SE 67%

A Preference-Driven Methodology for High-Quality Solidity Code Generation

Zhiyuan Peng, Xin Yin, Chenhao Ying, Chao Ni, Yuan Luo

专题命中偏好对齐：alignment(abstract)；DPO(abstract)

2505.13257 2025-09-30 cs.CL cs.AI cs.LG 67%

Is Active Persona Inference Necessary for Aligning Small Models to Personal Preferences?

Zilu Tang, Afra Feyza Akyürek, Ekin Akyürek, Derry Wijaya

机构 * Boston University（波士顿大学）； MIT（麻省理工学院）； Monash University Indonesia（墨尔本大学印度尼西亚分校）

专题命中偏好对齐：alignment(abstract)；分类 cs.CL、cs.AI、cs.LG

Comments 9 pages, EMNLP PALS workshop 2025

2509.22633 2025-09-29 stat.ML cs.AI cs.CL cs.LG math.ST stat.TH 67%

Towards Efficient Online Exploration for Reinforcement Learning with Human Feedback

Gen Li, Yuling Yan

机构 * Department of Statistics and Data Science, The Chinese University of Hong Kong（统计与数据科学系，香港中文大学）； Department of Statistics, University of Wisconsin-Madison（统计系，威斯康星大学麦迪逊分校）

专题命中偏好对齐：RLHF(abstract)；分类 cs.CL、cs.AI、cs.LG

2509.22281 2025-09-29 cs.CV cs.RO 67%

MesaTask: Towards Task-Driven Tabletop Scene Generation via 3D Spatial Reasoning

Jinkun Hao, Naifu Liang, Zhen Luo, Xudong Xu, Weipeng Zhong, Ran Yi, Yichen Jin, Zhaoyang Lyu, Feng Zheng, Lizhuang Ma, Jiangmiao Pang

机构 * Shanghai Jiao Tong University（上海交通大学）； Shanghai AI Laboratory（上海人工智能实验室）； SII Southern University of Science and Technology（南方科技大学）； Peking University（北京大学）

专题命中偏好对齐：alignment(abstract)；DPO(abstract)

Comments Accepted by NeurIPS 2025; Project page: https://mesatask.github.io/

2509.19249 2025-09-26 cs.CL cs.AI cs.LG 67%

Reinforcement Learning on Pre-Training Data

Siheng Li, Kejiao Li, Zenan Xu, Guanhua Huang, Evander Yang, Kun Li, Haoyuan Wu, Jiajia Wu, Zihao Zheng, Chenchen Zhang, Kun Shi, Kyrierl Deng, Qi Yi, Ruibin Xiong, Tingqiang Xu, Yuhao Jiang, Jianfeng Yan, Yuyuan Zeng, Guanghui Xu, Jinbao Xue, Zhijiang Xu, Zheng Fang, Shuai Li, Qibin Liu, Xiaoxue Li, Zhuoyu Li, Yangyu Tao, Fei Gao, Cheng Jiang, Bo Chao Wang, Kai Liu, Jianchen Zhu, Wai Lam, Wayyt Wang, Bo Zhou, Di Wang

机构 * LLM Department, Tencent（腾讯大模型部门）； HunYuan Infra Team（文心一言基础设施团队）； The Chinese University of Hong Kong（香港中文大学）

专题命中偏好对齐：RLHF(abstract)；分类 cs.CL、cs.AI、cs.LG

Comments Work in progress

2509.04897 2025-09-26 cs.CL cs.AI cs.LG 67%

PLaMo 2 Technical Report

Preferred Networks, :, Kaizaburo Chubachi, Yasuhiro Fujita, Shinichi Hemmi, Yuta Hirokawa, Kentaro Imajo, Toshiki Kataoka, Goro Kobayashi, Kenichi Maehashi, Calvin Metzger, Hiroaki Mikami, Shogo Murai, Daisuke Nishino, Kento Nozawa, Toru Ogawa, Shintarou Okada, Daisuke Okanohara, Shunta Saito, Shotaro Sano, Shuji Suzuki, Kuniyuki Takahashi, Daisuke Tanaka, Avinash Ummadisingu, Hanqin Wang, Sixue Wang, Tianqi Xu

机构 * Preferred Networks Inc（Preferred Networks公司）

专题命中偏好对齐：DPO(abstract)；分类 cs.CL、cs.AI、cs.LG

2502.12148 2025-09-26 cs.CV 67%

HermesFlow: Seamlessly Closing the Gap in Multimodal Understanding and Generation

Ling Yang, Xinchen Zhang, Ye Tian, Chenming Shang, Minghao Xu, Wentao Zhang, Bin Cui

机构 * Peking University（北京大学）； Tsinghua University（清华大学）； Mila - Québec AI Institute（魁北克人工智能研究所）

专题命中偏好对齐：alignment(abstract)；DPO(abstract)

Comments NeurIPS 2025. Code: https://github.com/Gen-Verse/HermesFlow

2505.13388 2025-09-23 cs.CL cs.AI cs.LG 67%

R3: Robust Rubric-Agnostic Reward Models

David Anugraha, Zilu Tang, Lester James V. Miranda, Hanyang Zhao, Mohammad Rifqi Farhansyah, Garry Kuwanto, Derry Wijaya, Genta Indra Winata

机构 * Stanford University（斯坦福大学）； Boston University（波士顿大学）； Columbia University（哥伦比亚大学）； University of Toronto（多伦多大学）； Institut Teknologi Bandung（Bandung 工程技术大学）； Monash University Indonesia（墨尔本大学印尼分校）； Capital One

专题命中偏好对齐：alignment(abstract)；分类 cs.CL、cs.AI、cs.LG

Comments Preprint

2508.15495 2025-09-18 cs.SE 67%

SynthCoder: A Synthetical Strategy to Tune LLMs for Code Completion

Dongjun Yu, Xiao Yan, Zhenrui Li, Jipeng Xiao, Haochuan He, Yongda Yu, Hao Zhang, Guoping Rong, Xiaobo Huang

专题命中偏好对齐：alignment(abstract)；DPO(abstract)

2509.08826 2025-09-11 cs.CV 67%

RewardDance: Reward Scaling in Visual Generation

Jie Wu, Yu Gao, Zilyu Ye, Ming Li, Liang Li, Hanzhong Guo, Jie Liu, Zeyue Xue, Xiaoxia Hou, Wei Liu, Yan Zeng, Weilin Huang

机构 * ByteDance Seed（字节跳动种子）

专题命中偏好对齐：alignment(abstract)；RLHF(abstract)

Comments Bytedance Seed Technical Report

2509.00685 2025-09-03 eess.AS cs.SD 67%

MPO: Multidimensional Preference Optimization for Language Model-based Text-to-Speech

Kangxiang Xia, Xinfa Zhu, Jixun Yao, Lei Xie

机构 * School of Computer Science, Northwestern Polytechnical University（计算机科学学院）

专题命中偏好对齐：alignment(abstract)；DPO(abstract)

Comments Accepted by NCMMSC2025

1. 偏好对齐 3273 篇

Closing the Gap: Data-Centric Fine-Tuning of Vision Language Models for the Standardized Exam Questions

EAGER: Edge-Aligned LLM Defense for Robust, Efficient, and Accurate Cybersecurity Question Answering

Preference Learning from Physics-Based Feedback: Tuning Language Models to Design BCC/B2 Superalloys

Reducing the Scope of Language Models

Interaction Dynamics as a Reward Signal for LLMs

Approximating Human Preferences Using a Multi-Judge Learned System

AutoLibra: Agent Metric Induction from Open-Ended Human Feedback

Preference Optimization by Estimating the Ratio of the Data Distribution

Better Estimation of the Kullback--Leibler Divergence Between Language Models

HelpSteer3-Preference: Open Human-Annotated Preference Data across Diverse Tasks and Languages

BLEUBERI: BLEU is a surprisingly effective reward for instruction following

The Narcissus Hypothesis: Descending to the Rung of Illusion

DenseDPO: Fine-Grained Temporal Preference Optimization for Video Diffusion Models

Investigating Thematic Patterns and User Preferences in LLM Interactions using BERTopic

Modality-Balancing Preference Optimization of Large Multimodal Models by Adversarial Negative Mining

Improving Neutral Point-of-View Generation with Data- and Parameter-Efficient RL

Out-of-Distribution Detection using Synthetic Data Generation

IMG: Calibrating Diffusion Models via Implicit Multimodal Guidance

From Faithfulness to Correctness: Generative Reward Models that Think Critically

A Preference-Driven Methodology for High-Quality Solidity Code Generation

Is Active Persona Inference Necessary for Aligning Small Models to Personal Preferences?

Towards Efficient Online Exploration for Reinforcement Learning with Human Feedback

MesaTask: Towards Task-Driven Tabletop Scene Generation via 3D Spatial Reasoning

Reinforcement Learning on Pre-Training Data

PLaMo 2 Technical Report

HermesFlow: Seamlessly Closing the Gap in Multimodal Understanding and Generation

R3: Robust Rubric-Agnostic Reward Models

SynthCoder: A Synthetical Strategy to Tune LLMs for Code Completion

RewardDance: Reward Scaling in Visual Generation

MPO: Multidimensional Preference Optimization for Language Model-based Text-to-Speech