大模型对齐与安全

Improving Multi-Step Reasoning Abilities of Large Language Models with Direct Advantage Policy Optimization

Jiacai Liu, Chaojie Wang, Chris Yuhao Liu, Liang Zeng, Rui Yan, Yiwen Sun, Yang Liu, Yahui Zhou

专题命中 安全训练 ：alignment(abstract)；DPO(abstract)；分类 cs.AI

Drawing the Line: Enhancing Trustworthiness of MLLMs Through the Power of Refusal

Yuhao Wang, Zhiyuan Zhu, Heyang Liu, Yusheng Liao, Hongcheng Liu, Yanfeng Wang, Yu Wang

专题命中 安全训练 ：alignment(abstract)；trustworthy(abstract)；分类 cs.CL

SCAR: Sparse Conditioned Autoencoders for Concept Detection and Steering in LLMs

Ruben Härle, Felix Friedrich, Manuel Brack, Björn Deiseroth, Patrick Schramowski, Kristian Kersting

专题命中 安全训练 ：alignment(abstract)；safety(abstract)；分类 cs.CL

Comments Accepted at Socially Responsible Language Modelling Research (SoLaR) Workshop at NeurIPS 2024

GPT-4o System Card

OpenAI, :, Aaron Hurst, Adam Lerer, Adam P. Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, Aleksander Mądry, Alex Baker-Whitcomb, Alex Beutel, Alex Borzunov, Alex Carney, Alex Chow, Alex Kirillov, Alex Nichol, Alex Paino, Alex Renzin, Alex Tachard Passos, Alexander Kirillov, Alexi Christakis, Alexis Conneau, Ali Kamali, Allan Jabri, Allison Moyer, Allison Tam, Amadou Crookes, Amin Tootoochian, Amin Tootoonchian, Ananya Kumar, Andrea Vallone, Andrej Karpathy, Andrew Braunstein, Andrew Cann, Andrew Codispoti, Andrew Galu, Andrew Kondrich, Andrew Tulloch, Andrey Mishchenko, Angela Baek, Angela Jiang, Antoine Pelisse, Antonia Woodford, Anuj Gosalia, Arka Dhar, Ashley Pantuliano, Avi Nayak, Avital Oliver, Barret Zoph, Behrooz Ghorbani, Ben Leimberger, Ben Rossen, Ben Sokolowsky, Ben Wang, Benjamin Zweig, Beth Hoover, Blake Samic, Bob McGrew, Bobby Spero, Bogo Giertler, Bowen Cheng, Brad Lightcap, Brandon Walkin, Brendan Quinn, Brian Guarraci, Brian Hsu, Bright Kellogg, Brydon Eastman, Camillo Lugaresi, Carroll Wainwright, Cary Bassin, Cary Hudson, Casey Chu, Chad Nelson, Chak Li, Chan Jun Shern, Channing Conger, Charlotte Barette, Chelsea Voss, Chen Ding, Cheng Lu, Chong Zhang, Chris Beaumont, Chris Hallacy, Chris Koch, Christian Gibson, Christina Kim, Christine Choi, Christine McLeavey, Christopher Hesse, Claudia Fischer, Clemens Winter, Coley Czarnecki, Colin Jarvis, Colin Wei, Constantin Koumouzelis, Dane Sherburn, Daniel Kappler, Daniel Levin, Daniel Levy, David Carr, David Farhi, David Mely, David Robinson, David Sasaki, Denny Jin, Dev Valladares, Dimitris Tsipras, Doug Li, Duc Phong Nguyen, Duncan Findlay, Edede Oiwoh, Edmund Wong, Ehsan Asdar, Elizabeth Proehl, Elizabeth Yang, Eric Antonow, Eric Kramer, Eric Peterson, Eric Sigler, Eric Wallace, Eugene Brevdo, Evan Mays, Farzad Khorasani, Felipe Petroski Such, Filippo Raso, Francis Zhang, Fred von Lohmann, Freddie Sulit, Gabriel Goh, Gene Oden, Geoff Salmon, Giulio Starace, Greg Brockman, Hadi Salman, Haiming Bao, Haitang Hu, Hannah Wong, Haoyu Wang, Heather Schmidt, Heather Whitney, Heewoo Jun, Hendrik Kirchner, Henrique Ponde de Oliveira Pinto, Hongyu Ren, Huiwen Chang, Hyung Won Chung, Ian Kivlichan, Ian O'Connell, Ian O'Connell, Ian Osband, Ian Silber, Ian Sohl, Ibrahim Okuyucu, Ikai Lan, Ilya Kostrikov, Ilya Sutskever, Ingmar Kanitscheider, Ishaan Gulrajani, Jacob Coxon, Jacob Menick, Jakub Pachocki, James Aung, James Betker, James Crooks, James Lennon, Jamie Kiros, Jan Leike, Jane Park, Jason Kwon, Jason Phang, Jason Teplitz, Jason Wei, Jason Wolfe, Jay Chen, Jeff Harris, Jenia Varavva, Jessica Gan Lee, Jessica Shieh, Ji Lin, Jiahui Yu, Jiayi Weng, Jie Tang, Jieqi Yu, Joanne Jang, Joaquin Quinonero Candela, Joe Beutler, Joe Landers, Joel Parish, Johannes Heidecke, John Schulman, Jonathan Lachman, Jonathan McKay, Jonathan Uesato, Jonathan Ward, Jong Wook Kim, Joost Huizinga, Jordan Sitkin, Jos Kraaijeveld, Josh Gross, Josh Kaplan, Josh Snyder, Joshua Achiam, Joy Jiao, Joyce Lee, Juntang Zhuang, Justyn Harriman, Kai Fricke, Kai Hayashi, Karan Singhal, Katy Shi, Kavin Karthik, Kayla Wood, Kendra Rimbach, Kenny Hsu, Kenny Nguyen, Keren Gu-Lemberg, Kevin Button, Kevin Liu, Kiel Howe, Krithika Muthukumar, Kyle Luther, Lama Ahmad, Larry Kai, Lauren Itow, Lauren Workman, Leher Pathak, Leo Chen, Li Jing, Lia Guy, Liam Fedus, Liang Zhou, Lien Mamitsuka, Lilian Weng, Lindsay McCallum, Lindsey Held, Long Ouyang, Louis Feuvrier, Lu Zhang, Lukas Kondraciuk, Lukasz Kaiser, Luke Hewitt, Luke Metz, Lyric Doshi, Mada Aflak, Maddie Simens, Madelaine Boyd, Madeleine Thompson, Marat Dukhan, Mark Chen, Mark Gray, Mark Hudnall, Marvin Zhang, Marwan Aljubeh, Mateusz Litwin, Matthew Zeng, Max Johnson, Maya Shetty, Mayank Gupta, Meghan Shah, Mehmet Yatbaz, Meng Jia Yang, Mengchao Zhong, Mia Glaese, Mianna Chen, Michael Janner, Michael Lampe, Michael Petrov, Michael Wu, Michele Wang, Michelle Fradin, Michelle Pokrass, Miguel Castro, Miguel Oom Temudo de Castro, Mikhail Pavlov, Miles Brundage, Miles Wang, Minal Khan, Mira Murati, Mo Bavarian, Molly Lin, Murat Yesildal, Nacho Soto, Natalia Gimelshein, Natalie Cone, Natalie Staudacher, Natalie Summers, Natan LaFontaine, Neil Chowdhury, Nick Ryder, Nick Stathas, Nick Turley, Nik Tezak, Niko Felix, Nithanth Kudige, Nitish Keskar, Noah Deutsch, Noel Bundick, Nora Puckett, Ofir Nachum, Ola Okelola, Oleg Boiko, Oleg Murk, Oliver Jaffe, Olivia Watkins, Olivier Godement, Owen Campbell-Moore, Patrick Chao, Paul McMillan, Pavel Belov, Peng Su, Peter Bak, Peter Bakkum, Peter Deng, Peter Dolan, Peter Hoeschele, Peter Welinder, Phil Tillet, Philip Pronin, Philippe Tillet, Prafulla Dhariwal, Qiming Yuan, Rachel Dias, Rachel Lim, Rahul Arora, Rajan Troll, Randall Lin, Rapha Gontijo Lopes, Raul Puri, Reah Miyara, Reimar Leike, Renaud Gaubert, Reza Zamani, Ricky Wang, Rob Donnelly, Rob Honsby, Rocky Smith, Rohan Sahai, Rohit Ramchandani, Romain Huet, Rory Carmichael, Rowan Zellers, Roy Chen, Ruby Chen, Ruslan Nigmatullin, Ryan Cheu, Saachi Jain, Sam Altman, Sam Schoenholz, Sam Toizer, Samuel Miserendino, Sandhini Agarwal, Sara Culver, Scott Ethersmith, Scott Gray, Sean Grove, Sean Metzger, Shamez Hermani, Shantanu Jain, Shengjia Zhao, Sherwin Wu, Shino Jomoto, Shirong Wu, Shuaiqi, Xia, Sonia Phene, Spencer Papay, Srinivas Narayanan, Steve Coffey, Steve Lee, Stewart Hall, Suchir Balaji, Tal Broda, Tal Stramer, Tao Xu, Tarun Gogineni, Taya Christianson, Ted Sanders, Tejal Patwardhan, Thomas Cunninghman, Thomas Degry, Thomas Dimson, Thomas Raoux, Thomas Shadwell, Tianhao Zheng, Todd Underwood, Todor Markov, Toki Sherbakov, Tom Rubin, Tom Stasi, Tomer Kaftan, Tristan Heywood, Troy Peterson, Tyce Walters, Tyna Eloundou, Valerie Qi, Veit Moeller, Vinnie Monaco, Vishal Kuo, Vlad Fomenko, Wayne Chang, Weiyi Zheng, Wenda Zhou, Wesam Manassra, Will Sheu, Wojciech Zaremba, Yash Patil, Yilei Qian, Yongjik Kim, Youlong Cheng, Yu Zhang, Yuchen He, Yuchen Zhang, Yujia Jin, Yunxing Dai, Yury Malkov

专题命中 安全训练 ：safety(abstract)；分类 cs.CL、cs.AI、cs.CY

ART: Automatic Red-teaming for Text-to-Image Models to Protect Benign Users

Guanlin Li, Kangjie Chen, Shudong Zhang, Jie Zhang, Tianwei Zhang

专题命中 安全训练 ：safety(abstract)；jailbreak(abstract)；分类 cs.AI

Comments Accepted by NeurIPS 2024

HiddenGuard: Fine-Grained Safe Generation with Specialized Representation Router

Lingrui Mei, Shenghua Liu, Yiwei Wang, Baolong Bi, Ruibin Yuan, Xueqi Cheng

专题命中 安全训练 ：alignment(abstract)；safety(abstract)；分类 cs.CL

Short-Long Policy Evaluation with Novel Actions

Hyunji Alex Nam, Yash Chandak, Emma Brunskill

专题命中 安全训练 ：safety(abstract)；AI safety(abstract)；分类 cs.LG

Comments Added references for related work

Towards Guaranteed Safe AI: A Framework for Ensuring Robust and Reliable AI Systems

David "davidad" Dalrymple, Joar Skalse, Yoshua Bengio, Stuart Russell, Max Tegmark, Sanjit Seshia, Steve Omohundro, Christian Szegedy, Ben Goldhaber, Nora Ammann, Alessandro Abate, Joe Halpern, Clark Barrett, Ding Zhao, Tan Zhi-Xuan, Jeannette Wing, Joshua Tenenbaum

专题命中 安全训练 ：safety(abstract)；AI safety(abstract)；分类 cs.AI

WildTeaming at Scale: From In-the-Wild Jailbreaks to (Adversarially) Safer Language Models

Liwei Jiang, Kavel Rao, Seungju Han, Allyson Ettinger, Faeze Brahman, Sachin Kumar, Niloofar Mireshghallah, Ximing Lu, Maarten Sap, Yejin Choi, Nouha Dziri

专题命中 安全训练 ：safety(abstract)；jailbreak(abstract)；分类 cs.CL

No Two Devils Alike: Unveiling Distinct Mechanisms of Fine-tuning Attacks

Chak Tou Leong, Yi Cheng, Kaishuai Xu, Jian Wang, Hanlin Wang, Wenjie Li

专题命中 安全训练 ：alignment(abstract)；safety(abstract)；分类 cs.CL

Comments work in progress

Unmasking and Improving Data Credibility: A Study with Datasets for Training Harmless Language Models

Zhaowei Zhu, Jialu Wang, Hao Cheng, Yang Liu

专题命中 安全训练 ：alignment(abstract)；分类 cs.CL、cs.AI、cs.CY

Comments ICLR 2024

In-Context Learning Can Re-learn Forbidden Tasks

Sophie Xhonneux, David Dobre, Jian Tang, Gauthier Gidel, Dhanya Sridhar

专题命中 安全训练 ：safety(abstract)；prompt injection(abstract)；分类 cs.LG

Comments 19 pages, 7 figures

Quantifying stability of non-power-seeking in artificial agents

Evan Ryan Gunter, Yevgeny Liokumovich, Victoria Krakovna

专题命中 安全训练 ：alignment(abstract)；safety(abstract)；分类 cs.AI

Comments 37 pages, 5 figures

Knowledge Augmented Machine Learning with Applications in Autonomous Driving: A Survey

Julian Wörmann, Daniel Bogdoll, Christian Brunner, Etienne Bührle, Han Chen, Evaristus Fuh Chuo, Kostadin Cvejoski, Ludger van Elst, Philip Gottschall, Stefan Griesche, Christian Hellert, Christian Hesels, Sebastian Houben, Tim Joseph, Niklas Keil, Johann Kelsch, Mert Keser, Hendrik Königshof, Erwin Kraft, Leonie Kreuser, Kevin Krone, Tobias Latka, Denny Mattern, Stefan Matthes, Franz Motzkus, Mohsin Munir, Moritz Nekolla, Adrian Paschke, Stefan Pilar von Pilchau, Maximilian Alexander Pintz, Tianming Qiu, Faraz Qureishi, Syed Tahseen Raza Rizvi, Jörg Reichardt, Laura von Rueden, Alexander Sagel, Diogo Sasdelli, Tobias Scholl, Gerhard Schunk, Gesina Schwalbe, Hao Shen, Youssef Shoeb, Hendrik Stapelbroek, Vera Stehr, Gurucharan Srinivas, Anh Tuan Tran, Abhishek Vivekanandan, Ya Wang, Florian Wasserrab, Tino Werner, Christian Wirth, Stefan Zwicklbauer

专题命中 安全训练 ：safety(abstract)；trustworthy(abstract)；分类 cs.LG

Comments 111 pages, Added section on Run-time Network Verification

Frontier Language Models are not Robust to Adversarial Arithmetic, or "What do I need to say so you agree 2+2=5?

C. Daniel Freeman, Laura Culp, Aaron Parisi, Maxwell L Bileschi, Gamaleldin F Elsayed, Alex Rizkowsky, Isabelle Simpson, Alex Alemi, Azade Nova, Ben Adlam, Bernd Bohnet, Gaurav Mishra, Hanie Sedghi, Igor Mordatch, Izzeddin Gur, Jaehoon Lee, JD Co-Reyes, Jeffrey Pennington, Kelvin Xu, Kevin Swersky, Kshiteej Mahajan, Lechao Xiao, Rosanne Liu, Simon Kornblith, Noah Constant, Peter J. Liu, Roman Novak, Yundi Qian, Noah Fiedel, Jascha Sohl-Dickstein

专题命中 安全训练 ：alignment(abstract)；分类 cs.CL、cs.AI、cs.CY

A Risk-Sensitive Approach to Policy Optimization

Jared Markowitz, Ryan W. Gardner, Ashley Llorens, Raman Arora, I-Jeng Wang

专题命中 安全训练 ：safety(abstract)；AI safety(abstract)；分类 cs.LG

Comments 16 pages, 13 figures. AAAI 2023 (Special Track on Safe and Robust AI)

Prompt Packer: Deceiving LLMs through Compositional Instruction with Hidden Attacks

Shuyu Jiang, Xingshu Chen, Rui Tang

专题命中 安全训练 ：alignment(abstract)；safety(abstract)；分类 cs.CL

Absolutist AI

Mitchell Barrington

专题命中 安全训练 ：safety(abstract)；AI safety(abstract)；分类 cs.AI

Collective Reasoning for Safe Autonomous Systems

Selma Saidi

专题命中 安全训练 ：safety(abstract)；trustworthy(abstract)；分类 cs.CY

Safer Conversational AI as a Source of User Delight

Xiaoding Lu, Aleksey Korshuk, Zongyi Liu, William Beauchamp, Chai Research

专题命中 安全训练 ：safety(abstract)；AI safety(abstract)；分类 cs.AI

Conditioning Predictive Models: Risks and Strategies

Evan Hubinger, Adam Jermyn, Johannes Treutlein, Rubi Hudson, Kate Woolverton

专题命中 安全训练 ：alignment(abstract)；safety(abstract)；分类 cs.AI

Model-based Safe Deep Reinforcement Learning via a Constrained Proximal Policy Optimization Algorithm

Ashish Kumar Jayant, Shalabh Bhatnagar

专题命中 安全训练 ：safety(abstract)；AI safety(abstract)；分类 cs.LG

Comments Proceedings of NeurIPS 2022

Novelty Detection via Network Saliency in Visual-based Deep Learning

Valerie Chen, Man-Ki Yoon, Zhong Shao

专题命中 安全训练 ：safety(abstract)；trustworthy(abstract)；分类 cs.LG

Comments To be published in Dependable and Secure Machine Learning (DSML) workshop co-located with the IEEE Conference on Dependable Systems and Networks 2019

Philosophy-informed Machine Learning

MZ Naser

专题命中 安全训练 ：alignment(abstract)；分类 cs.AI、cs.CY、cs.LG

Mechanist: AI as a Scientific Instrument for Discovering the Mechanisms of Intelligence

Mechanist：作为科学仪器的AI，用于发现智能的机制

Mengru Wang, Junfeng Fang, Shuofei Qiao, Zhenqian Xu, Haoming Xu, Haoxiong Wang, Shumin Deng, Linyi Yang, Zhixiang Cui, Xin Xu, Yunzhi Yao, Buqiang Xu, Fei Shen, Haozhe Luo, Yunxiang Wei, Ningyu Zhang, Julian McAuley, Tat Seng Chua, Huajun Chen

机构 * Zhejiang University（浙江大学） ； National University of Singapore（新加坡国立大学） ； Heriot-Watt University（赫瑞瓦特大学） ； Southern University of Science and Technology（南方科技大学） ； University of California, San Diego（加利福尼亚大学圣迭戈分校） ； Northeastern University（东北大学） 纠错

专题命中 安全训练 ：safety(abstract)；分类 cs.CL、cs.AI、cs.LG

AI总结 本研究推出智能体系统Mechanist，以AI为科学仪器自主发现AI智能机制，其生成假设更有价值、实验更可靠，还能发现安全风险、构建信念机制理论并转化为提升模型性能的干预措施。

Comments Work in progress

Multi-Agent Embodied Autonomous Driving (MAEAD): From V2X Information Exchange to Shared World Models

多智能体具身自动驾驶：从V2X信息交换到共享世界模型

Senkang Hu, Zhengru Fang, Yihang Tao, Zihan Fang, Yiqin Deng, Yuguang Fang

机构 * Lingnan University, Hong Kong（岭南大学（香港）） 纠错

专题命中 安全训练 ：alignment(abstract)；safety(abstract)

AI总结 本文综述了从单车智能向多智能体具身系统转变的自动驾驶技术，通过共享世界模型实现感知共享、意图推断和协同规划，并指出了在仿真评估、实时安全保证等方面的研究空白。

Agentic Transaction: Towards ACID-Compliant Agent Systems

智能体事务：面向ACID兼容的智能体系统

Zhaoyan Sun, Xiaoxiao Wang, Guoliang Li

专题命中 安全训练 ：trustworthy(abstract)；分类 cs.CL、cs.AI、cs.LG

AI总结 该研究提出ACID兼容的智能体事务框架，开发对应数据智能体，在基准测试中较含Claude Code的现有智能体提升10.6%，为构建可信可扩展AI智能体开辟新方向。

SomaliBench Eval: Measuring English-to-Somali Refusal Gaps in Open-Weight Language Models

SomaliBench Eval：衡量开源语言模型中英语到索马里语的拒绝差距

Khalid Yusuf Dahir

机构 * Independent researcher（独立研究人员） 纠错

专题命中 安全训练 ：safety(abstract)；分类 cs.CL、cs.AI、cs.CY

AI总结 通过构建索马里语有害意图基准并评估四个开源模型，发现英语到索马里语的拒绝率存在显著差距，且多数非拒绝输出为不流畅的无效内容。

Comments v2: prose rewritten; classification-pipeline correction (34 judge-declined rows manually labeled); paired bootstrap and McNemar statistics; LLM-use disclosure added

Nuclear fusion for AI: A pathway to power data centers sustainably

AI的核聚变：可持续为数据中心供电的途径

Layla Araiinejad, Vineet Jagadeesan Nair

专题命中 安全训练 ：alignment(abstract)；safety(abstract)

AI总结 该研究探讨核聚变能否为AI驱动型数据中心提供可持续电源，通过技术经济分析发现其适配数据中心需求，具备成本竞争力与可行性，应优先部署。

Comments Under review

VirtualCrime: Evaluating the Criminal Potential and Agentic Behaviors of Large Language Models via Sandbox Simulation

VirtualCrime: 通过沙盒模拟评估大语言模型的犯罪潜力

Yilin Tang, Yu Wang, Lanlan Qiu, Wenchang Gao, Yunfei Ma, Baicheng Chen, Tianxing He

专题命中 安全训练 ：alignment(abstract)；safety(abstract)

AI总结 本文提出VirtualCrime框架，通过三代理系统评估大语言模型的犯罪能力，设计40种多样化的犯罪任务，并评估8种强大的LLM，发现模型在犯罪过程中生成详细计划并执行智能犯罪过程，但有时会采取严重行动伤害NPC。