大模型对齐与安全

AGrail: A Lifelong Agent Guardrail with Effective and Adaptive Safety Detection

Weidi Luo, Shenghong Dai, Xiaogeng Liu, Suman Banerjee, Huan Sun, Muhao Chen, Chaowei Xiao

专题命中 安全训练 ：safety(title,abstract)；分类 cs.AI

How Safe is Your Safety Metric? Automatic Concatenation Tests for Metric Reliability

Ora Nova Fandina, Leshem Choshen, Eitan Farchi, George Kour, Yotam Perlitz, Orna Raz

专题命中 安全训练 ：safety(title,abstract)；分类 cs.AI

Gen-AI for User Safety: A Survey

Akshar Prabhu Desai, Tejasvi Ravi, Mohammad Luqman, Mohit Sharma, Nithya Kota, Pranjul Yadav

专题命中 安全训练 ：safety(title,abstract)；分类 cs.AI

Journal ref 2024 IEEE International Conference on Big Data (BigData), Washington, DC, USA, 2024, pp. 5315-5324

Why human-AI relationships need socioaffective alignment

Hannah Rose Kirk, Iason Gabriel, Chris Summerfield, Bertie Vidgen, Scott A. Hale

专题命中 安全训练 ：alignment(title,abstract)；分类 cs.AI

A sketch of an AI control safety case

Tomek Korbak, Joshua Clymer, Benjamin Hilton, Buck Shlegeris, Geoffrey Irving

专题命中 安全训练 ：safety(title,abstract)；分类 cs.AI

ASTRAL: Automated Safety Testing of Large Language Models

Miriam Ugarte, Pablo Valle, José Antonio Parejo, Sergio Segura, Aitor Arrieta

专题命中 安全训练 ：safety(title,abstract)；分类 cs.CL

Journal ref The 6th ACM/IEEE International Conference on Automation of Software Test (AST 2025)

MSTS: A Multimodal Safety Test Suite for Vision-Language Models

Paul Röttger, Giuseppe Attanasio, Felix Friedrich, Janis Goldzycher, Alicia Parrish, Rishabh Bhardwaj, Chiara Di Bonaventura, Roman Eng, Gaia El Khoury Geagea, Sujata Goswami, Jieun Han, Dirk Hovy, Seogyeong Jeong, Paloma Jeretič, Flor Miriam Plaza-del-Arco, Donya Rooein, Patrick Schramowski, Anastassia Shaitarova, Xudong Shen, Richard Willats, Andrea Zugarini, Bertie Vidgen

专题命中 安全训练 ：safety(title,abstract)；分类 cs.CL

Comments under review

Safety through feedback in Constrained RL

Shashank Reddy Chirra, Pradeep Varakantham, Praveen Paruchuri

专题命中 安全训练 ：safety(title,abstract)；分类 cs.AI

Comments Accepted at NeurIPS 2024 (Poster)

Autonomous Alignment with Human Value on Altruism through Considerate Self-imagination and Theory of Mind

Haibo Tong, Enmeng Lu, Yinqian Sun, Zhengqiang Han, Chao Liu, Feifei Zhao, Yi Zeng

专题命中 安全训练 ：alignment(title)；safety(abstract)；分类 cs.AI

Safe LoRA: the Silver Lining of Reducing Safety Risks when Fine-tuning Large Language Models

Chia-Yi Hsu, Yu-Lin Tsai, Chih-Hsun Lin, Pin-Yu Chen, Chia-Mu Yu, Chun-Ying Huang

专题命中 安全训练 ：safety(title,abstract)；分类 cs.LG

Comments This is the camera-ready version accepted for NeurIPS 2024

SafeWatch: An Efficient Safety-Policy Following Video Guardrail Model with Transparent Explanations

Zhaorun Chen, Francesco Pinto, Minzhou Pan, Bo Li

专题命中 安全训练 ：safety(title,abstract)；分类 cs.LG

Comments 43 pages, 20 figures

Alignment at Pre-training! Towards Native Alignment for Arabic LLMs

Juhao Liang, Zhenyang Cai, Jianqing Zhu, Huang Huang, Kewei Zong, Bang An, Mosen Alharthi, Juncai He, Lian Zhang, Haizhou Li, Benyou Wang, Jinchao Xu

专题命中 安全训练 ：alignment(title,abstract)；分类 cs.CL

Comments Accepted to NeurIPS 2024 main conference. see https://github.com/FreedomIntelligence/AceGPT-v2

Action-Attentive Deep Reinforcement Learning for Autonomous Alignment of Beamlines

Siyu Wang, Shengran Dai, Jianhui Jiang, Shuang Wu, Yufei Peng, Junbin Zhang

专题命中 安全训练 ：alignment(title,abstract)；分类 cs.LG

Comments 17 pages, 5 figures

CoCA: Regaining Safety-awareness of Multimodal Large Language Models with Constitutional Calibration

Jiahui Gao, Renjie Pi, Tianyang Han, Han Wu, Lanqing Hong, Lingpeng Kong, Xin Jiang, Zhenguo Li

专题命中 安全训练 ：safety(title,abstract)；分类 cs.CL

Comments 10 pages, COLM-2024

Mapping Technical Safety Research at AI Companies: A literature review and incentives analysis

Oscar Delaney, Oliver Guest, Zoe Williams

专题命中 安全训练 ：safety(title,abstract)；分类 cs.CY

Comments Sep-25 update adds papers that we missed in the original dataset, as well as more acknowledgements

Enhancing RL Safety with Counterfactual LLM Reasoning

Dennis Gross, Helge Spieker

专题命中 安全训练 ：safety(title,abstract)；分类 cs.LG

Safety Constrained Multi-Agent Reinforcement Learning for Active Voltage Control

Yang Qu, Jinming Ma, Feng Wu

专题命中 安全训练 ：safety(title,abstract)；分类 cs.LG

Comments Accepted by IJCAI2024

Compromesso! Italian Many-Shot Jailbreaks Undermine the Safety of Large Language Models

Fabio Pernisi, Dirk Hovy, Paul Röttger

专题命中 安全训练 ：safety(title,abstract)；分类 cs.CL

Comments Accepted at ACL 2024 (Student Research Workshop)

A Safe Exploration Strategy for Model-free Task Adaptation in Safety-constrained Grid Environments

Erfan Entezami, Mahsa Sahebdel, Dhawal Gupta

专题命中 安全训练 ：safety(title,abstract)；分类 cs.AI

Adapting Safe-for-Work Classifier for Malaysian Language Text: Enhancing Alignment in LLM-Ops Framework

Aisyah Razak, Ariff Nazhan, Kamarul Adha, Wan Adzhar Faiq Adzlan, Mas Aisyah Ahmad, Ammar Azman

专题命中 安全训练 ：alignment(title,abstract)；分类 cs.CL

Reinforcement Learning in a Safety-Embedded MDP with Trajectory Optimization

Fan Yang, Wenxuan Zhou, Zuxin Liu, Ding Zhao, David Held

专题命中 安全训练 ：safety(title,abstract)；分类 cs.AI

Safe-Embed: Unveiling the Safety-Critical Knowledge of Sentence Encoders

Jinseok Kim, Jaewon Jung, Sangyeop Kim, Sohyung Park, Sungzoon Cho

专题命中 安全训练 ：safety(title,abstract)；分类 cs.CL

Comments ACL 2024 KnowledgeableLMs workshop paper

Enhancing Safety for Autonomous Agents in Partly Concealed Urban Traffic Environments Through Representation-Based Shielding

Pierre Haritz, David Wanke, Thomas Liebig

专题命中 安全训练 ：safety(title,abstract)；分类 cs.LG

ROSE Doesn't Do That: Boosting the Safety of Instruction-Tuned Large Language Models with Reverse Prompt Contrastive Decoding

Qihuang Zhong, Liang Ding, Juhua Liu, Bo Du, Dacheng Tao

专题命中 安全训练 ：safety(title,abstract)；分类 cs.CL

Comments Accepted to ACL2024 Findings

Information Theoretic Guarantees For Policy Alignment In Large Language Models

Youssef Mroueh

专题命中 安全训练 ：alignment(title,abstract)；分类 cs.LG

ISAACS: Iterative Soft Adversarial Actor-Critic for Safety

Kai-Chieh Hsu, Duy Phuong Nguyen, Jaime Fernández Fisac

专题命中 安全训练 ：safety(title,abstract)；分类 cs.LG

Comments Accepted in 5th Annual Learning for Dynamics & Control Conference (L4DC), University of Pennsylvania

Dishonesty in Helpful and Harmless Alignment

Youcheng Huang, Jingkun Tang, Duanyu Feng, Zheng Zhang, Wenqiang Lei, Jiancheng Lv, Anthony G. Cohn

专题命中 安全训练 ：alignment(title,abstract)；分类 cs.CL

Safety through Permissibility: Shield Construction for Fast and Safe Reinforcement Learning

Alexander Politowicz, Sahisnu Mazumder, Bing Liu

专题命中 安全训练 ：safety(title,abstract)；分类 cs.LG

Comments 9 pages, 3 figures

MoGU: A Framework for Enhancing Safety of Open-Sourced LLMs While Preserving Their Usability

Yanrui Du, Sendong Zhao, Danyang Zhao, Ming Ma, Yuhan Chen, Liangyu Huo, Qing Yang, Dongliang Xu, Bing Qin

专题命中 安全训练 ：safety(title,abstract)；分类 cs.CL

Safety in Graph Machine Learning: Threats and Safeguards

Song Wang, Yushun Dong, Binchi Zhang, Zihan Chen, Xingbo Fu, Yinhan He, Cong Shen, Chuxu Zhang, Nitesh V. Chawla, Jundong Li

专题命中 安全训练 ：safety(title,abstract)；分类 cs.LG

Comments 20 pages