大模型对齐与安全

Exploring the Vulnerability of the Content Moderation Guardrail in Large Language Models via Intent Manipulation

Jun Zhuang, Haibo Jin, Ye Zhang, Zhengjian Kang, Wenbin Zhang, Gaby G. Dagher, Haohan Wang

机构 * Boise State University（博伊州立大学） ； University of Illinois Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校） ； University of Pittsburgh（匹兹堡大学） ； New York University（纽约大学） ； Florida International University（佛罗里达国际大学） 纠错

专题命中 安全训练 ：safety(abstract)；jailbreak(abstract)；分类 cs.CL、cs.AI

Comments Accepted for EMNLP'25 Findings. TL;DR: We propose a new two-stage intent-based prompt-refinement framework, IntentPrompt, that aims to explore the vulnerability of LLMs' content moderation guardrails by refining prompts into benign-looking declarative forms via intent manipulation for red-teaming purposes

Personalized Constitutionally-Aligned Agentic Superego: Secure AI Behavior Aligned to Diverse Human Values

Nell Watson, Ahmed Amer, Evan Harris, Preeti Ravindra, Shujun Zhang

专题命中 安全训练 ：alignment(abstract)；safety(abstract)；分类 cs.AI、cs.CY

Comments 42 pages, 6 figures

Journal ref Information 2025, 16(8), 651

Just Enough Shifts: Mitigating Over-Refusal in Aligned Language Models with Targeted Representation Fine-Tuning

Mahavir Dabas, Si Chen, Charles Fleming, Ming Jin, Ruoxi Jia

机构 * Department of Electrical and Computer Engineering, Virginia Tech, Blacksburg, VA, USA（弗吉尼亚理工学院与州立大学电气与计算机工程系） ； Cisco Research, San Jose, CA, USA（思科研究） 纠错

专题命中 安全训练 ：alignment(abstract)；safety(abstract)；分类 cs.AI、cs.LG

OR-Bench: An Over-Refusal Benchmark for Large Language Models

Justin Cui, Wei-Lin Chiang, Ion Stoica, Cho-Jui Hsieh

专题命中 安全训练 ：alignment(abstract)；safety(abstract)；分类 cs.CL、cs.AI

Comments Accepted to ICML 2025, we thank everyone for their valuable suggestions and feedback!

Constructive Symbolic Reinforcement Learning via Intuitionistic Logic and Goal-Chaining Inference

Andrei T. Patrascu

机构 * FAST Foundation（FAST基金会） 纠错

专题命中 安全训练 ：safety(abstract)；trustworthy(abstract)；分类 cs.AI、cs.LG

Ensuring Safe and High-Quality Outputs: A Guideline Library Approach for Language Models

Yi Luo, Zhenghao Lin, Yuhao Zhang, Jiashuo Sun, Chen Lin, Chengjin Xu, Xiangdong Su, Yelong Shen, Jian Guo, Yeyun Gong

机构 * XiaMen University（厦门大学） ； Yanshan University（雁山大学） ； IDEA Research（IDEA研究） ； Inner Mongolia University（内蒙古大学） ； Microsoft（微软） ； Microsoft Research Asia（微软亚洲研究院） 纠错

专题命中 安全训练 ：alignment(abstract)；safety(abstract)；分类 cs.CL、cs.AI

Comments Accepted to NAACL 2024 main conference

Journal ref Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers)

Test-Time Immunization: A Universal Defense Framework Against Jailbreaks for (Multimodal) Large Language Models

Yongcan Yu, Yanbo Wang, Ran He, Jian Liang

机构 * NLPR & MAIS, Institute of Automation, Chinese Academy of Sciences（国家工程实验室与人工智能研究所，中国科学院） ； School of Artificial Intelligence, University of Chinese Academy of Sciences（中国科学院大学人工智能学院） 纠错

专题命中 安全训练 ：safety(abstract)；jailbreak(abstract)；分类 cs.CL、cs.AI

Comments Under Review

A Flexible Large Language Models Guardrail Development Methodology Applied to Off-Topic Prompt Detection

Gabriel Chua, Shing Yee Chan, Shaun Khoo

专题命中 安全训练 ：safety(abstract)；jailbreak(abstract)；分类 cs.CL、cs.LG

Comments 8 pages, 5 figures

Which Information should the UK and US AISI share with an International Network of AISIs? Opportunities, Risks, and a Tentative Proposal

Lara Thurnherr

专题命中 安全训练 ：safety(abstract)；AI safety(abstract)；分类 cs.AI、cs.CY

Comments 12 Pages, 3 Tables, 2 Figures

On Calibration of LLM-based Guard Models for Reliable Content Moderation

Hongfu Liu, Hengguan Huang, Xiangming Gu, Hao Wang, Ye Wang

专题命中 安全训练 ：safety(abstract)；jailbreak(abstract)；分类 cs.CL、cs.LG

Comments Accepted to ICLR 2025

C3AI: Crafting and Evaluating Constitutions for Constitutional AI

Yara Kyrychenko, Ke Zhou, Edyta Bogucka, Daniele Quercia

专题命中 安全训练 ：alignment(abstract)；safety(abstract)；分类 cs.CL、cs.AI

Comments This has been accepted for the Web Conference 2025

What AI evaluations for preventing catastrophic risks can and cannot do

Peter Barnett, Lisa Thiergart

专题命中 安全训练 ：safety(abstract)；AI safety(abstract)；分类 cs.AI、cs.CY

Semantic Loss Guided Data Efficient Supervised Fine Tuning for Safe Responses in LLMs

Yuxiao Lu, Arunesh Sinha, Pradeep Varakantham

专题命中 安全训练 ：alignment(abstract)；safety(abstract)；分类 cs.CL、cs.AI

HonestLLM: Toward an Honest and Helpful Large Language Model

Chujie Gao, Siyuan Wu, Yue Huang, Dongping Chen, Qihui Zhang, Zhengyan Fu, Yao Wan, Lichao Sun, Xiangliang Zhang

专题命中 安全训练 ：alignment(abstract)；trustworthy(abstract)；分类 cs.CL、cs.AI

Can an AI Agent Safely Run a Government? Existence of Probably Approximately Aligned Policies

Frédéric Berdoz, Roger Wattenhofer

专题命中 安全训练 ：alignment(abstract)；safety(abstract)；分类 cs.AI、cs.CY

Comments Accepted at NeurIPS 2024

Breach By A Thousand Leaks: Unsafe Information Leakage in `Safe' AI Responses

David Glukhov, Ziwen Han, Ilia Shumailov, Vardan Papyan, Nicolas Papernot

专题命中 安全训练 ：alignment(abstract)；safety(abstract)；分类 cs.AI、cs.CY

Refusing Safe Prompts for Multi-modal Large Language Models

Zedian Shao, Hongbin Liu, Yuepeng Hu, Neil Zhenqiang Gong

专题命中 安全训练 ：alignment(abstract)；safety(abstract)；分类 cs.AI、cs.LG

Safe-CLIP: Removing NSFW Concepts from Vision-and-Language Models

Samuele Poppi, Tobia Poppi, Federico Cocchi, Marcella Cornia, Lorenzo Baraldi, Rita Cucchiara

专题命中 安全训练 ：safety(abstract)；trustworthy(abstract)；分类 cs.CL、cs.AI

Comments ECCV 2024

Towards Interpretable Reinforcement Learning with Constrained Normalizing Flow Policies

Finn Rietz, Erik Schaffernicht, Stefan Heinrich, Johannes A. Stork

专题命中 安全训练 ：alignment(abstract)；safety(abstract)；分类 cs.AI、cs.LG

Evaluating Large Language Models: A Comprehensive Survey

Zishan Guo, Renren Jin, Chuang Liu, Yufei Huang, Dan Shi, Supryadi, Linhao Yu, Yan Liu, Jiaxuan Li, Bojian Xiong, Deyi Xiong

专题命中 安全训练 ：alignment(abstract)；safety(abstract)；分类 cs.CL、cs.AI

Comments 111 pages

Conformal Prediction with Large Language Models for Multi-Choice Question Answering

Bhawesh Kumar, Charlie Lu, Gauri Gupta, Anil Palepu, David Bellamy, Ramesh Raskar, Andrew Beam

专题命中 安全训练 ：safety(abstract)；trustworthy(abstract)；分类 cs.CL、cs.LG

Comments Updated sections on prompt engineering. Expanded sections 4.1 and 4.2 and appendix. Included additional references. Work published at the ICML 2023 (Neural Conversational AI TEACH) workshop

Appropriateness is all you need!

Hendrik Kempt, Alon Lavie, Saskia K. Nagel

专题命中 安全训练 ：alignment(abstract)；safety(abstract)；分类 cs.CL、cs.AI

Emerging Threats in Deep Learning-Based Autonomous Driving: A Comprehensive Survey

Hui Cao, Wenlong Zou, Yinkun Wang, Ting Song, Mengjun Liu

专题命中 安全训练 ：safety(abstract)；trustworthy(abstract)；分类 cs.AI、cs.LG

Comments 28 pages,10 figures

Parenting: Safe Reinforcement Learning from Human Input

Christopher Frye, Ilya Feige

专题命中 安全训练 ：safety(abstract)；AI safety(abstract)；分类 cs.AI、cs.LG

Comments 9 pages, 4 figures, 1 table

Constitution or Collapse? Exploring Constitutional AI with Llama 3-8B

Xue Zhang

专题命中 安全训练 ：DPO(abstract)；harmlessness(abstract)；分类 cs.AI；alignment(comments)

Comments 6 pages, 2 figures. Conducted as part of research on alignment techniques for language models

Emergent Problem-Graph Alignment in RL-Discovered Entanglement Topologies for QAOA

用于QAOA的RL发现纠缠拓扑中的涌现问题-图对齐

Tobias Rohe, Federico Harjes Ruiloba, Markus Baumann, Gerhard Stenzel, Leo Sünkel, Thomas Gabor, Claudia Linnhoff-Popien

专题命中 安全训练 ：alignment(title)

AI总结 该研究用RL智能体发现QAOA的纠缠拓扑，得到与问题图对齐的稀疏拓扑，在有限优化预算下性能优于全图拓扑，揭示了拓扑密度带来的可训练性与表达性的权衡。

MIND-V: Hierarchical World Model for Long-Horizon Robotic Manipulation with RL-based Physical Alignment

MIND-V：基于强化学习物理对齐的长期机器人操作分层世界模型

Ruicheng Zhang, Mingyang Zhang, Jun Zhou, Xiaofan Liu, Zunnan Xu, Zhizhou Zhong, Puxin Yan, Haocheng Luo, Xiu Li

机构 * Tsinghua University（清华大学） ； X Square Robot（X Square机器人） ； Sun Yat-sen University（中山大学） ； HKUST（香港科技大学） 纠错

专题命中 安全训练 ：alignment(title)

AI总结 提出MIND-V分层世界模型，通过语义推理、行为语义桥接和运动视频生成，结合强化学习物理对齐，实现长期机器人操作视频的物理合理合成。

Stable but Unsafe: Agent-Driven Cyber-Physical Systems Under Gain Manipulation Attacks

无安全性的稳定性：对自主网络物理系统的增益操纵攻击

Ali Eslami, Jiangbo Yu

专题命中 安全训练 ：safety(title)

AI总结 本文形式化自主网络物理系统中的增益操纵攻击，通过三轴攻击模型和分类法，揭示稳定性保持的增益替换仍可产生远超安全限度的瞬态放大，并利用Bauer-Fike特征值界和Kreiss矩阵定理推导隐蔽条件和最坏影响。

Comments 6 pages, 2 figures, 2 tables. Submitted to IEEE L-CSS for possible publication

ForceForget: Reinforcement Concept Removal for Enhancing Safety in Text-to-Image Models

ForceForget: 通过强化概念移除增强文本到图像模型的安全性

Dong Han, Yong Li

机构 * Dong Han（董汉） ； Yong Li（李勇） 纠错

专题命中 安全训练 ：safety(title)

AI总结 针对文本到图像模型生成不安全内容的问题，提出基于强化学习优化概念擦除奖励的方法，通过安全适配器调节文本嵌入，在消除不安全内容的同时保持模型对安全语义的生成能力。

Comments Accepted to ICML 2026

Beyond Semantic Dominance: Cognitive Affective Reasoning and Empathetic Response Alignment in Audio Language Models

超越语义主导：音频语言模型中的认知情感推理与共情响应对齐

Zhixian Zhao, Shuiyuan Wang, Wenjie Tian, Jingbin Hu, Ziyu Zhang, Lei Xie

机构 * Northwestern Polytechnical University（西北工业大学） 纠错

专题命中 安全训练 ：alignment(title)

AI总结 提出CogAudio-LLM框架，通过构建LIME-440K数据集实现声学-语义解耦，设计EIPS思维链机制进行心理推理，并采用DR-SAPO优化策略平衡逻辑严谨性与共情质量，解决音频语言模型中的语义主导和情感认知不足问题。

Comments Accepted by Interspeech2026