ToolSword: Unveiling Safety Issues of Large Language Models in Tool Learning Across Three Stages
School of Computer Science, Fudan University(复旦大学计算机科学学院);Shanghai Collaborative Innovation Center of Intelligent Visual Computing(上海智能视觉计算协同创新中心);Institute of Modern Languages and Linguistics, Fudan University(复旦大学现代语言与语言学研究所)
Junjie Ye, Sixian Li, Guanyu Li, Caishuang Huang, Songyang Gao, Yilong Wu, Qi Zhang, Tao Gui, Xuanjing Huang
Tool learning is widely acknowledged as a foundational approach or deploying large language models (LLMs) in real-world scenarios. While current research primarily emphasizes leveraging tools to augment LLMs, it frequently neglects emerging safety considerations tied to their application. To fill this gap, we present *ToolSword*, a comprehensive framework dedicated to meticulously investigating safety issues linked to LLMs in tool learning. Specifically, ToolSword delineates six safety scenarios for LLMs in tool learning, encompassing **malicious queries** and **jailbreak attacks** in the input stage, **noisy misdirection** and **risky cues** in the execution stage, and **harmful feedback** and **error conflicts** in the output stage. Experiments conducted on 11 open-source and closed-source LLMs reveal enduring safety challenges in tool learning, such as handling harmful queries, employing risky tools, and delivering detrimental feedback, which even GPT-4 is susceptible to. Moreover, we conduct further studies with the aim of fostering research on tool learning safety. The data is released in https://github.com/Junjie-Ye/ToolSword.
Understanding and Mitigating Overrefusal in LLMs from an Unveiling Perspective of Safety Decision Boundary
The State Key Laboratory of Blockchain and Data Security(区块链与数据安全国家重点实验室);Hangzhou High-Tech Zone (Binjiang) Institute of Blockchain and Data Security(杭州高新技术区(滨江)区块链与数据安全研究院);Language and Machine Intelligence Department, Ant Group(蚂蚁集团语言与机器智能部门)
ProjLens: Unveiling the Role of Projectors in Multimodal Model Safety
University of Science and Technology of China(中国科学技术大学);Beijing University of Aeronautics and Astronautics(北京航空航天大学);Nanyang Technological University(南洋理工大学);Southwest Jiaotong University(西南交通大学);Shanghai Artificial Intelligence Laboratory(上海人工智能实验室)
Jailbreak-as-a-Service++: Unveiling Distributed AI-Driven Malicious Information Campaigns Powered by LLM Crowdsourcing
Institute of Computing Technology, CAS(中国科学院计算技术研究所);University of Chinese Academy of Sciences(中国科学院大学);China People’s Public Security University(中国人民公安大学);Tongji University(同济大学);South China Normal University(华南师范大学);Tsing Hua University(清华大学)
Unveiling Trust in Multimodal Large Language Models: Evaluation, Analysis, and Mitigation
Department of Computer Science and Technology, College of AI, Institute for AI, Tsinghua-Bosch Joint ML Center, THBI Lab, BNRist Center, Tsinghua University(计算机科学与技术系、人工智能学院、人工智能研究所、清华-博世联合机器学习中心、THBI实验室、BNRist中心、清华大学);Institute of Artificial Intelligence, Beihang University(人工智能研究院、北航);RealAI
When Models Outthink Their Safety: Unveiling and Mitigating Self-Jailbreak in Large Reasoning Models
Chinese Information Processing Laboratory, Institute of Software, Chinese Academy of Sciences(中国科学院软件研究所信息处理实验室);University of Chinese Academy of Sciences(中国科学院大学)
Think in Safety: Unveiling and Mitigating Safety Alignment Collapse in Multimodal Large Reasoning Model
Key Laboratory of Big Data & Artificial Intelligence in Transportation (Beijing Jiaotong University), Ministry of Education(大数据与人工智能交通联合实验室(北京交通大学));School of Computer Science and Technology, Beijing Jiaotong University(计算机科学与技术学院,北京交通大学);Tsinghua University(清华大学)
The Missing Half: Unveiling Training-time Implicit Safety Risks Beyond Deployment
The Conversational AI (CoAI) group, DCST, Tsinghua University(清华大学智能对话研究院);National University of Singapore(新加坡国立大学);Pattern Recognition Center, WeChat AI, Tencent Inc, China(腾讯人工智能研究院)