arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2402.10753cs.CLcs.AI
收藏

ToolSword:揭示大语言模型在工具学习三个阶段中的安全问题

ToolSword: Unveiling Safety Issues of Large Language Models in Tool Learning Across Three Stages

School of Computer Science, Fudan University(复旦大学计算机科学学院);Shanghai Collaborative Innovation Center of Intelligent Visual Computing(上海智能视觉计算协同创新中心);Institute of Modern Languages and Linguistics, Fudan University(复旦大学现代语言与语言学研究所)

Junjie Ye, Sixian Li, Guanyu Li, Caishuang Huang, Songyang Gao, Yilong Wu, Qi Zhang, Tao Gui, Xuanjing Huang

AI总结:

本研究提出ToolSword框架,从输入、执行、输出三个阶段划分六种安全场景,揭示11款开源及闭源大语言模型在工具学习中普遍存在安全漏洞,以推动相关安全研究。

摘要:

工具学习被公认为是在现实场景中部署大语言模型(LLMs)的基础方法。当前研究主要强调利用工具增强LLMs,却常常忽略了与其应用相关的新兴安全考量。为填补这一空白,我们提出ToolSword,这是一个专门用于细致研究LLMs在工具学习中相关安全问题的综合框架。具体而言,ToolSword划分了LLMs在工具学习中的六种安全场景,包括输入阶段的恶意查询与越狱攻击、执行阶段的噪声误导与风险提示、输出阶段的有害反馈与错误冲突。我们对11个开源和闭源LLM开展了实验,结果揭示了工具学习中存在的持续性安全挑战,例如处理有害查询、使用风险工具以及提供有害反馈,即便GPT-4也易受此类问题影响。此外,我们还开展了进一步研究,旨在推动工具学习安全领域的研究。相关数据已发布于https://github.com/Junjie-Ye/ToolSword。

英文摘要:

Tool learning is widely acknowledged as a foundational approach or deploying large language models (LLMs) in real-world scenarios. While current research primarily emphasizes leveraging tools to augment LLMs, it frequently neglects emerging safety considerations tied to their application. To fill this gap, we present *ToolSword*, a comprehensive framework dedicated to meticulously investigating safety issues linked to LLMs in tool learning. Specifically, ToolSword delineates six safety scenarios for LLMs in tool learning, encompassing **malicious queries** and **jailbreak attacks** in the input stage, **noisy misdirection** and **risky cues** in the execution stage, and **harmful feedback** and **error conflicts** in the output stage. Experiments conducted on 11 open-source and closed-source LLMs reveal enduring safety challenges in tool learning, such as handling harmful queries, employing risky tools, and delivering detrimental feedback, which even GPT-4 is susceptible to. Moreover, we conduct further studies with the aim of fostering research on tool learning safety. The data is released in https://github.com/Junjie-Ye/ToolSword.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑