大模型对齐与安全

HumorReject: Decoupling LLM Safety from Refusal Prefix via A Little Humor

Zihui Wu, Haichang Gao, Jiacheng Luo, Zhaoxiang Liu

专题命中 安全训练 ：safety(title,abstract)；alignment(abstract)；分类 cs.LG

SafeEditor: Unified MLLM for Efficient Post-hoc T2I Safety Editing

Ruiyang Zhang, Jiahao Luo, Xiaoru Feng, Qiufan Pang, Yaodong Yang, Juntao Dai

机构 * PKU Alignment Team, Peking University（北京大学对齐团队） ； LLM Safety Centre, Beijing Academy of Artificial Intelligence（北京人工智能研究院大语言模型安全中心） 纠错

专题命中 安全训练 ：safety(title,abstract)；alignment(abstract)；分类 cs.AI

A Guardrail for Safety Preservation: When Safety-Sensitive Subspace Meets Harmful-Resistant Null-Space

Bingjie Zhang, Yibo Yang, Zhe Ren, Dandan Guo, Jindong Gu, Philip Torr, Bernard Ghanem

机构 * School of Artificial Intelligence, Jilin University（吉林大学人工智能学院） ； King Abdullah University of Science and Technology（卡塔尔国王大学科学与技术研究院） ； University of Oxford（牛津大学） 纠错

专题命中 安全训练 ：safety(title,abstract)；alignment(abstract)；分类 cs.AI

Steering Over-refusals Towards Safety in Retrieval Augmented Generation

Utsav Maskey, Mark Dras, Usman Naseem

机构 * Macquarie University（麦觉里大学） 纠错

专题命中 安全训练 ：safety(title,abstract)；alignment(abstract)；分类 cs.CL

Comments Preprint

Safety-Aligned Weights Are Not Enough: Refusal-Teacher-Guided Finetuning Enhances Safety and Downstream Performance under Harmful Finetuning Attacks

Seokil Ham, Yubin Choi, Yujin Yang, Seungju Cho, Younghun Kim, Changick Kim

机构 * Korea Advanced Institute of Science and Technology (KAIST)（韩国科学技术院） 纠错

专题命中 安全训练 ：safety(title,abstract)；alignment(abstract)；分类 cs.CL

When Safe Unimodal Inputs Collide: Optimizing Reasoning Chains for Cross-Modal Safety in Multimodal Large Language Models

Wei Cai, Shujuan Liu, Jian Zhao, Ziyan Shi, Yusheng Zhao, Yuchen Yuan, Tianle Zhang, Chi Zhang, Xuelong Li

专题命中 安全训练 ：safety(title,abstract)；alignment(abstract)；分类 cs.AI

Safety Pretraining: Toward the Next Generation of Safe AI

Pratyush Maini, Sachin Goyal, Dylan Sam, Alex Robey, Yash Savani, Yiding Jiang, Andy Zou, Matt Fredrikson, Zacharcy C. Lipton, J. Zico Kolter

专题命中 安全训练 ：safety(title,abstract)；alignment(abstract)；分类 cs.LG

Large Language Model's Multi-Capability Alignment in Biomedical Domain

Wentao Wu, Linqing Chen, Hanmeng Zhong, Weilei Wang

专题命中 安全训练 ：alignment(title,abstract)；safety(abstract)；分类 cs.AI

Measuring What Matters: A Framework for Evaluating Safety Risks in Real-World LLM Applications

Jia Yi Goh, Shaun Khoo, Nyx Iskandar, Gabriel Chua, Leanne Tan, Jessica Foo

专题命中 安全训练 ：safety(title,abstract)；AI safety(abstract)；分类 cs.CY

Multi-modal Representations for Fine-grained Multi-label Critical View of Safety Recognition

Britty Baby, Vinkle Srivastav, Pooja P. Jain, Kun Yuan, Pietro Mascagni, Nicolas Padoy

机构 * University of Strasbourg, CNRS, INSERM, ICube, UMR7357, Strasbourg, France（斯特拉斯堡大学，法国国家科学研究中心，法国国家卫生研究院，ICube，UMR7357，斯特拉斯堡，法国） ； Fondazione Policlinico Universitario A. Gemelli IRCCS, Università Cattolica del Sacro Cuore, Rome, Italy（A. Gemelli IRCCS大学医院，罗马，意大利） ； CAMP, Technische Universität München, Munich, Germany（慕尼黑技术大学，德国） ； Institute of Image-Guided Surgery, IHU Strasbourg, Strasbourg, France（影像引导手术研究所，斯特拉斯堡，法国） 纠错

专题命中 安全训练 ：safety(title,abstract)；alignment(abstract)；分类 cs.AI

The Structural Safety Generalization Problem

Julius Broomfield, Tom Gibbs, Ethan Kosak-Hine, George Ingebretsen, Tia Nasir, Jason Zhang, Reihaneh Iranmanesh, Sara Pieri, Reihaneh Rabbany, Kellin Pelrine

机构 * Georgia Tech（佐治亚理工学院） ； Mila ； UC Berkeley（加州大学伯克利分校） ； Stanford（斯坦福大学） ； MBZUAI（穆桑大学人工智能研究所） ； McGill（麦吉尔大学） 纠错

专题命中 安全训练 ：safety(title,abstract)；AI safety(abstract)；分类 cs.AI

A Framework for Benchmarking and Aligning Task-Planning Safety in LLM-Based Embodied Agents

Yuting Huang, Leilei Ding, Zhipeng Tang, Tianfu Wang, Xinrui Lin, Wuyang Zhang, Mingxiao Ma, Yanyong Zhang

机构 * University of Science and Technology of China（中国科学技术大学） 纠错

专题命中 安全训练 ：safety(title,abstract)；alignment(abstract)；分类 cs.AI

Comments 16 pages, 10 figures

CURATe: Benchmarking Personalised Alignment of Conversational AI Assistants

Lize Alberts, Benjamin Ellis, Andrei Lupu, Jakob Foerster

专题命中 安全训练 ：alignment(title,abstract)；safety(abstract)；分类 cs.AI

Dynamic safety cases for frontier AI

Carmen Cârlan, Francesca Gomez, Yohan Mathew, Ketana Krishna, René King, Peter Gebauer, Ben R. Smith

专题命中 安全训练 ：safety(title,abstract)；alignment(abstract)；分类 cs.CY

Comments 75 pages, 41 tables/figures

SCANS: Mitigating the Exaggerated Safety for LLMs via Safety-Conscious Activation Steering

Zouying Cao, Yifei Yang, Hai Zhao

专题命中 安全训练 ：safety(title,abstract)；alignment(abstract)；分类 cs.AI

Comments Extended version of paper accepted to AAAI 2025. 14 pages, 6 figures

WildGuard: Open One-Stop Moderation Tools for Safety Risks, Jailbreaks, and Refusals of LLMs

Seungju Han, Kavel Rao, Allyson Ettinger, Liwei Jiang, Bill Yuchen Lin, Nathan Lambert, Yejin Choi, Nouha Dziri

专题命中 安全训练 ：safety(title,abstract)；jailbreak(abstract)；分类 cs.CL

Comments NeurIPS 2024 Camera Ready. First two authors contributed equally. Third and fourth authors contributed equally

Self and Cross-Model Distillation for LLMs: Effective Methods for Refusal Pattern Alignment

Jie Li, Yi Liu, Chongyang Liu, Xiaoning Ren, Ling Shi, Weisong Sun, Yinxing Xue

专题命中 安全训练 ：alignment(title,abstract)；RLHF(abstract)；分类 cs.CL

Comments The method used in the paper has obvious problems and ambiguities. The security enhancement method we used cannot be considered distillation, but it is described as distillation in the paper, and the experiment lacks comparison and baseline, which has been criticized by many peers. In order to avoid further dissemination, we have decided to withdraw the paper

Safe to Serve: Aligning Instruction-Tuned Models for Safety and Helpfulness

Avinash Amballa, Durga Sandeep Saluru, Gayathri Akkinapalli, Abhishek Sureddy, Akshay Kumar Sureddy

专题命中 安全训练 ：safety(title,abstract)；DPO(abstract)；分类 cs.CL

Comments 18 pages

Locking Down the Finetuned LLMs Safety

Minjun Zhu, Linyi Yang, Yifan Wei, Ningyu Zhang, Yue Zhang

专题命中 安全训练 ：safety(title,abstract)；alignment(abstract)；分类 cs.CL

Mitigating Exaggerated Safety in Large Language Models

Ruchira Ray, Ruchi Bhalani

专题命中 安全训练 ：safety(title,abstract)；jailbreak(abstract)；分类 cs.CL

Comments 17 pages, 8 figures, 2 tables

Is it safe to cross? Interpretable Risk Assessment with GPT-4V for Safety-Aware Street Crossing

Hochul Hwang, Sunjae Kwon, Yekyung Kim, Donghyun Kim

专题命中 安全训练 ：safety(title,abstract)；trustworthy(abstract)；分类 cs.AI

Safety Fine-Tuning at (Almost) No Cost: A Baseline for Vision Large Language Models

Yongshuo Zong, Ondrej Bohdal, Tingyang Yu, Yongxin Yang, Timothy Hospedales

专题命中 安全训练 ：safety(title,abstract)；alignment(abstract)；分类 cs.LG

Comments ICML 2024

MLLM-Protector: Ensuring MLLM's Safety without Hurting Performance

Renjie Pi, Tianyang Han, Jianshu Zhang, Yueqi Xie, Rui Pan, Qing Lian, Hanze Dong, Jipeng Zhang, Tong Zhang

专题命中 安全训练 ：safety(title,abstract)；alignment(abstract)；分类 cs.CL

Safety-Tuned LLaMAs: Lessons From Improving the Safety of Large Language Models that Follow Instructions

Federico Bianchi, Mirac Suzgun, Giuseppe Attanasio, Paul Röttger, Dan Jurafsky, Tatsunori Hashimoto, James Zou

专题命中 安全训练 ：safety(title,abstract)；harmlessness(abstract)；分类 cs.CL

SimpleSafetyTests: a Test Suite for Identifying Critical Safety Risks in Large Language Models

Bertie Vidgen, Nino Scherrer, Hannah Rose Kirk, Rebecca Qian, Anand Kannappan, Scott A. Hale, Paul Röttger

专题命中 安全训练 ：safety(title,abstract)；AI safety(abstract)；分类 cs.CL

MART: Improving LLM Safety with Multi-round Automatic Red-Teaming

Suyu Ge, Chunting Zhou, Rui Hou, Madian Khabsa, Yi-Chia Wang, Qifan Wang, Jiawei Han, Yuning Mao

专题命中 安全训练 ：safety(title,abstract)；alignment(abstract)；分类 cs.CL

Towards a framework for evaluating the safety, acceptability and efficacy of AI systems for health: an initial synthesis

Jessica Morley, Caroline Morton, Kassandra Karpathakis, Mariarosaria Taddeo, Luciano Floridi

专题命中 安全训练 ：safety(title,abstract)；AI safety(abstract)；分类 cs.AI

Frictive Policy Optimization for LLMs: Epistemic Intervention, Risk-Sensitive Control, and Reflective Alignment

摩擦性政策优化用于大语言模型：认知干预、风险敏感控制与反思对齐

James Pustejovsky, Nikhil Krishnaswamy

机构 * Brandeis University（布拉德雷大学） ； Colorado State University（科罗拉多州立大学） 纠错

专题命中 安全训练 ：alignment(title,abstract)；分类 cs.CL、cs.AI、cs.LG

AI总结 本文提出摩擦性政策优化框架，通过干预管理认知与规范风险，引入摩擦干预分类和统一方法，提升模型的认知能力与对齐性。

Comments Frictive Policy Optimization; epistemic alignment; risk-sensitive control; LLM alignment; clarification and refusal; preference learning; trust regions; dialogue agents

Being Considerate as a Pathway Towards Pluralistic Alignment for Agentic AI

Parand A. Alamdari, Toryn Q. Klassen, Rodrigo Toro Icarte, Sheila A. McIlraith

专题命中 安全训练 ：alignment(title,abstract)；分类 cs.AI、cs.CY、cs.LG

Comments Pluralistic Alignment Workshop at NeurIPS 2024

DUET: Dual-Teacher On-Policy Distillation via Same-Weight Disagreement for Prohibition Compliance

DUET：基于同权重分歧的双教师在线策略蒸馏用于禁止合规性

Zihan Li, Feifei Li, Wenhui Que

专题命中 安全训练 ：DPO(abstract,abstract_cn)；alignment(abstract)；safety(abstract)；分类 cs.CL、cs.LG

AI总结 本研究针对LLM部署中的动态禁止规则合规问题，提出DUET双教师在线策略蒸馏方法，构建工业基准，在Qwen模型上实现高合规性与效用保留，性能优于基线。