arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2406.10802cs.CLcs.AI
收藏

KGPA:基于跨领域知识图谱的大语言模型鲁棒性评估

KGPA: Robustness Evaluation for Large Language Models via Cross-Domain Knowledge Graphs

Waseda University(早稻田大学);Southeast University(东南大学);Wuhan University(武汉大学)

Aihua Pei, Zehua Yang, Shunan Zhu, Ruoxi Cheng, Ju Jia, Lina Wang

AI总结:

本文提出KGPA框架,利用跨领域知识图谱生成对抗性提示,系统评估大语言模型的鲁棒性,并揭示了ChatGPT系列模型鲁棒性排序及专业领域对鲁棒性的影响。

摘要:

现有的大语言模型(LLMs)鲁棒性评估框架过度依赖特定基准,增加了成本,并且由于数据集限制,无法评估LLMs在专业领域的表现。本文提出了一个框架,利用知识图谱(KGs)系统地评估LLMs在对抗攻击场景下的鲁棒性。我们的框架从知识图谱的三元组中生成原始提示,并通过投毒方式创建对抗性提示,通过对抗攻击的结果来评估LLMs的鲁棒性。我们系统地评估了该框架及其模块的有效性。实验表明,ChatGPT系列的对抗鲁棒性排序为GPT-4-turbo > GPT-4o > GPT-3.5-turbo,且大语言模型的鲁棒性受到其运行所在专业领域的影响。

英文摘要:

Existing frameworks for assessing robustness of large language models (LLMs) overly depend on specific benchmarks, increasing costs and failing to evaluate performance of LLMs in professional domains due to dataset limitations. This paper proposes a framework that systematically evaluates the robustness of LLMs under adversarial attack scenarios by leveraging knowledge graphs (KGs). Our framework generates original prompts from the triplets of knowledge graphs and creates adversarial prompts by poisoning, assessing the robustness of LLMs through the results of these adversarial attacks. We systematically evaluate the effectiveness of this framework and its modules. Experiments show that adversarial robustness of the ChatGPT family ranks as GPT-4-turbo > GPT-4o > GPT-3.5-turbo, and the robustness of large language models is influenced by the professional domains in which they operate.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑