大模型对齐与安全

PsyLite Technical Report

Fangjun Ding, Renyu Zhang, Xinyu Feng, Chengye Xie, Zheng Zhang, Yanting Zhang

机构 * Donghua University（东华大学） 纠错

专题命中 偏好对齐 ：safety(abstract)；分类 cs.AI

Enhancing Safe and Controllable Protein Generation via Knowledge Preference Optimization

Yuhao Wang, Keyan Ding, Kehua Feng, Zeyuan Wang, Ming Qin, Xiaotong Li, Qiang Zhang, Huajun Chen

机构 * Zhejiang University（浙江大学） ； ZJU-Hangzhou Global Scientific and Technological Innovation Center（浙大杭州国际科创中心） 纠错

专题命中 偏好对齐 ：safety(abstract)；分类 cs.AI

Comments Accepted at ACL 2025 (Main Conference)

Best-of-N through the Smoothing Lens: KL Divergence and Regret Analysis

Gholamali Aminian, Idan Shenfeld, Amir R. Asadi, Ahmad Beirami, Youssef Mroueh

机构 * The Alan Turing Institute, London, UK（艾伦·图灵研究所） ； Statistical Laboratory, University of Cambridge, Cambridge, UK（剑桥大学统计实验室） ； Massachusetts Institute of Technology, USA（麻省理工学院） ； IBM Research, USA（IBM研究院） 纠错

专题命中 偏好对齐 ：alignment(abstract)；分类 cs.LG

Comments Workshop on Efficient Systems for Foundation Models at iCML

Differentiable Reward Optimization for LLM based TTS system

Changfeng Gao, Zhihao Du, Shiliang Zhang

机构 * Speech Team, Tongyi Lab, Alibaba Group（通义实验室，阿里巴巴集团） 纠错

专题命中 偏好对齐 ：RLHF(abstract)；分类 cs.AI

Generalist Reward Models: Found Inside Large Language Models

Yi-Chen Li, Tian Xu, Yang Yu, Xuqin Zhang, Xiong-Hui Chen, Zhongxiang Ling, Ningjing Chao, Lei Yuan, Zhi-Hua Zhou

机构 * National Key Laboratory for Novel Software Technology, Nanjing University, China（国家新型软件技术重点实验室，南京大学，中国） ； School of Artificial Intelligence, Nanjing University, China（人工智能学院，南京大学，中国） 纠错

专题命中 偏好对齐 ：alignment(abstract)；分类 cs.CL

Enhancing User Engagement in Socially-Driven Dialogue through Interactive LLM Alignments

Jiashuo Wang, Kaitao Song, Chunpu Xu, Changhe Song, Yang Xiao, Dongsheng Li, Lili Qiu, Wenjie Li

机构 * Department of Computing, The Hong Kong Polytechnic University（香港理工大学计算机系） ； Microsoft Research Asia（微软亚洲研究院） 纠错

专题命中 偏好对齐 ：DPO(abstract)；分类 cs.CL

Do Large Language Models Advocate for Inferentialism?

Yuzuki Arai, Sho Tsugawa

机构 * College of Media Arts, Science and Technology, School of Informatics, University of Tsukuba（筑波大学媒体艺术、科学与技术学院，信息学院） ； Institute of Systems and Information Engineering, University of Tsukuba（筑波大学系统与信息工程研究所） 纠错

专题命中 偏好对齐 ：RLHF(abstract)；分类 cs.CL

LettinGo: Explore User Profile Generation for Recommendation System

Lu Wang, Di Zhang, Fangkai Yang, Pu Zhao, Jianfeng Liu, Yuefeng Zhan, Hao Sun, Qingwei Lin, Weiwei Deng, Dongmei Zhang, Feng Sun, Qi Zhang

机构 * Microsoft Corporation（微软公司） ； Peking University（北京大学） 纠错

专题命中 偏好对齐 ：DPO(abstract)；分类 cs.AI

Comments 11 pages, 3 figures

From RAG to Agentic: Validating Islamic-Medicine Responses with LLM Agents

Mohammad Amaan Sayeed, Mohammed Talha Alam, Raza Imam, Shahab Saquib Sohail, Amir Hussain

机构 * Mohamed bin Zayed University of Artificial Intelligence, UAE（阿布扎克穆罕默德·本·扎耶德人工智能大学） ； Edinburgh Napier University, UK（爱丁堡纳皮尔大学） ； VIT Bhopal University, India（比哈尔大学） 纠错

专题命中 偏好对齐 ：safety(abstract)；分类 cs.CL

Comments Published at the 4th Muslims in Machine Learning (MusIML) Workshop (ICML-25)

Robust Reward Modeling via Causal Rubrics

Pragya Srivastava, Harman Singh, Rahul Madhavan, Gandharv Patil, Sravanti Addepalli, Arun Suggala, Rengarajan Aravamudhan, Soumya Sharma, Anirban Laha, Aravindan Raghuveer, Karthikeyan Shanmugam, Doina Precup

专题命中 偏好对齐 ：safety(abstract)；分类 cs.LG

RadFabric: Agentic AI System with Reasoning Capability for Radiology

Wenting Chen, Yi Dong, Zhaojun Ding, Yucheng Shi, Yifan Zhou, Fang Zeng, Yijun Luo, Tianyu Lin, Yihang Su, Yichen Wu, Kai Zhang, Zhen Xiang, Tianming Liu, Ninghao Liu, Lichao Sun, Yixuan Yuan, Xiang Li

专题命中 偏好对齐 ：alignment(abstract)；分类 cs.CL

Comments 4 figures, 2 tables

VL-GenRM: Enhancing Vision-Language Verification via Vision Experts and Iterative Training

Jipeng Zhang, Kehao Miao, Renjie Pi, Zhaowei Wang, Runtao Liu, Rui Pan, Tong Zhang

机构 * The Hong Kong University of Science and Technology（香港科学与技术大学） ； Nanyang Technological University（南洋理工大学） ； University of Illinois Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校） 纠错

专题命中 偏好对齐 ：alignment(abstract)；分类 cs.CL

REPA: Russian Error Types Annotation for Evaluating Text Generation and Judgment Capabilities

Alexander Pugachev, Alena Fenogenova, Vladislav Mikhailov, Ekaterina Artemova

机构 * HSE University（俄罗斯高等经济学院） ； SaluteDevices ； University of Oslo（奥斯陆大学） ； Toloka AI 纠错

专题命中 偏好对齐 ：alignment(abstract)；分类 cs.CL

Comments To appear at SIGSLAV 2025

Fino1: On the Transferability of Reasoning-Enhanced LLMs and Reinforcement Learning to Finance

Lingfei Qian, Weipeng Zhou, Yan Wang, Xueqing Peng, Han Yi, Yilun Zhao, Jimin Huang, Qianqian Xie, Jian-yun Nie

机构 * The FinAI（FinAI） ； Georgia Institute of Technology（佐治亚理工学院） ； Yale University（耶鲁大学） ； University of Montreal（蒙特利尔大学） 纠错

专题命中 偏好对齐 ：DPO(abstract)；分类 cs.CL

Comments 13 pages, 2 figures, 3 Tables

Step-Audio-AQAA: a Fully End-to-End Expressive Large Audio Language Model

Ailin Huang, Bingxin Li, Bruce Wang, Boyong Wu, Chao Yan, Chengli Feng, Heng Wang, Hongyu Zhou, Hongyuan Wang, Jingbei Li, Jianjian Sun, Joanna Wang, Mingrui Chen, Peng Liu, Ruihang Miao, Shilei Jiang, Tian Fei, Wang You, Xi Chen, Xuerui Yang, Yechang Huang, Yuxiang Zhang, Zheng Ge, Zheng Gong, Zhewei Huang, Zixin Zhang, Bin Wang, Bo Li, Buyun Ma, Changxin Miao, Changyi Wan, Chen Xu, Dapeng Shi, Dingyuan Hu, Enle Liu, Guanzhe Huang, Gulin Yan, Hanpeng Hu, Haonan Jia, Jiahao Gong, Jiaoren Wu, Jie Wu, Jie Yang, Junzhe Lin, Kaixiang Li, Lei Xia, Longlong Gu, Ming Li, Nie Hao, Ranchen Ming, Shaoliang Pang, Siqi Liu, Song Yuan, Tiancheng Cao, Wen Li, Wenqing He, Xu Zhao, Xuelin Zhang, Yanbo Yu, Yinmin Zhong, Yu Zhou, Yuanwei Liang, Yuanwei Lu, Yuxiang Yang, Zidong Yang, Zili Zhang, Binxing Jiao, Heung-Yeung Shum, Jiansheng Chen, Jing Li, Xiangyu Zhang, Xinhao Zhang, Yibo Zhu, Daxin Jiang, Shuchang Zhou, Chen Hu

机构 * Step-Audio Team（Step-Audio团队） 纠错

专题命中 偏好对齐 ：DPO(abstract)；分类 cs.CL

Comments 12 pages, 3 figures

EQA-RM: A Generative Embodied Reward Model with Test-time Scaling

Yuhang Chen, Zhen Tan, Tianlong Chen

机构 * The University of North Carolina at Chapel Hill（北卡罗来纳大学教堂山分校） ； Arizona State University（亚利桑那州立大学） 纠错

专题命中 偏好对齐 ：alignment(abstract)；分类 cs.LG

Comments preprint

Self-Steering Optimization: Autonomous Preference Optimization for Large Language Models

Hao Xiang, Bowen Yu, Hongyu Lin, Keming Lu, Yaojie Lu, Xianpei Han, Ben He, Le Sun, Jingren Zhou, Junyang Lin

专题命中 偏好对齐 ：alignment(abstract)；分类 cs.CL

MEMETRON: Metaheuristic Mechanisms for Test-time Response Optimization of Large Language Models

Son The Nguyen, Theja Tulabandhula

机构 * Department of Information and Decision Sciences University of Illinois Chicago（信息与决策科学系伊利诺伊大学香槟分校） 纠错

专题命中 偏好对齐 ：alignment(abstract)；分类 cs.CL

Multi-Step Visual Reasoning with Visual Tokens Scaling and Verification

Tianyi Bai, Zengjie Hu, Fupeng Sun, Jiantao Qiu, Yizhen Jiang, Guangxin He, Bohan Zeng, Conghui He, Binhang Yuan, Wentao Zhang

机构 * HKUST（香港科技大学） ； Peking University（北京大学） ； Shanghai AI Lab（上海人工智能实验室） ； Imperial College London（伦敦帝国理工学院） 纠错

专题命中 偏好对齐 ：DPO(abstract)；分类 cs.CL

From Axioms to Algorithms: Mechanized Proofs of the vNM Utility Theorem

Li Jingyuan

机构 * Department of Operations and Risk Management, Lingnan University（岭南大学运营与风险管理系） 纠错

专题命中 偏好对齐 ：alignment(abstract)；分类 cs.AI

WiNGPT-3.0 Technical Report

Boqin Zhuang, Chenxiao Song, Huitong Lu, Jiacheng Qiao, Mingqian Liu, Mingxing Yu, Ping Hong, Rui Li, Xiaoxia Song, Xiangjun Xu, Xu Chen, Yaoyao Ma, Yujie Gao

机构 * WiNGPT Team（WiNGPT团队） 纠错

专题命中 偏好对齐 ：trustworthy(abstract)；分类 cs.CL

Self-Correction is More than Refinement: A Learning Framework for Visual and Language Reasoning Tasks

Jiayi He, Hehai Lin, Qingyun Wang, Yi Fung, Heng Ji

机构 * University of Illinois at Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校） ； Georgia Institute of Technology（佐治亚理工学院） ； The Hong Kong University of Science and Technology（香港科技大学） 纠错

专题命中 偏好对齐 ：DPO(abstract)；分类 cs.CL

Comments Accepted by ACL 2025 Findings

SuperWriter: Reflection-Driven Long-Form Generation with Large Language Models

Yuhao Wu, Yushi Bai, Zhiqiang Hu, Juanzi Li, Roy Ka-Wei Lee

专题命中 偏好对齐 ：DPO(abstract)；分类 cs.CL

BPO: Revisiting Preference Modeling in Direct Preference Optimization

Lin Sun, Chuang Liu, Peng Liu, Bingyang Li, Weijia Lu, Ning Wu

专题命中 偏好对齐 ：DPO(abstract)；分类 cs.CL

Leveraging Sparsity for Sample-Efficient Preference Learning: A Theoretical Perspective

Yunzhen Yao, Lie He, Michael Gastpar

机构 * LINX, EPFL, Lausanne, Switzerland（日内瓦EPFL实验室） ； School of Computing and Artificial Intelligence, Shanghai University of Finance and Economics, Shanghai, China（上海金融学院计算机与人工智能学院） 纠错

专题命中 偏好对齐 ：alignment(abstract)；分类 cs.LG

Iterative Self-Improvement of Vision Language Models for Image Scoring and Self-Explanation

Naoto Tanji, Toshihiko Yamasaki

专题命中 偏好对齐 ：alignment(abstract)；分类 cs.CL

Comments Accepted to ICIP2025

Afterburner: Reinforcement Learning Facilitates Self-Improving Code Efficiency Optimization

Mingzhe Du, Luu Anh Tuan, Yue Liu, Yuhao Qing, Dong Huang, Xinyi He, Qian Liu, Zejun Ma, See-kiong Ng

专题命中 偏好对齐 ：DPO(abstract)；分类 cs.AI

GAS: Generative Auto-bidding with Post-training Search

Yewen Li, Shuai Mao, Jingtong Gao, Nan Jiang, Yunjian Xu, Qingpeng Cai, Fei Pan, Peng Jiang, Bo An

机构 * Nanyang Technological University（南洋理工大学） ； The Chinese University of Hong Kong（香港中文大学） ； City University of Hong Kong（香港城市大学） ； Kuaishou Technology（快手科技） 纠错

专题命中 偏好对齐 ：alignment(abstract)；分类 cs.AI

Contextual Candor: Enhancing LLM Trustworthiness Through Hierarchical Unanswerability Detection

Steven Robinson, Antonio Carlos Rivera

机构 * EDP University of Puerto Rico: San Sebastian（EDP巴尔的摩大学：圣塞巴斯蒂安） 纠错

专题命中 偏好对齐 ：RLHF(abstract)；分类 cs.CL

HSCR: Hierarchical Self-Contrastive Rewarding for Aligning Medical Vision Language Models

Songtao Jiang, Yan Zhang, Yeying Jin, Zhihang Tang, Yangyang Wu, Yang Feng, Jian Wu, Zuozhu Liu

机构 * Zhejiang University（浙江大学） ； Byte Dance（字节跳动） ； National University of Singapore（新加坡国立大学） ； Angelalign Inc China（Angelalign中国公司） ； Zhejiang Key Laboratory of Medical Imaging Artificial Intelligence（浙江省医学影像人工智能重点实验室） 纠错

专题命中 偏好对齐 ：alignment(abstract)；分类 cs.CL