SelfPrompt：通过领域约束知识指南和精炼对抗性提示自主评估大语言模型鲁棒性
SelfPrompt: Autonomously Evaluating LLM Robustness via Domain-Constrained Knowledge Guidelines and Refined Adversarial Prompts
AI总结：
针对传统LLM鲁棒性评估依赖标准化基准导致成本高、领域受限的问题，提出SelfPrompt框架，利用领域约束知识图谱生成精炼对抗性提示，让LLM自主评估鲁棒性，在多模型测试中减少传统数据依赖并实现高效针对性评估。
摘要：
传统的大语言模型（LLMs）鲁棒性评估方法常依赖标准化基准，这会增加成本并限制跨不同领域的评估。本文提出一种新框架，通过整合精炼对抗性提示和知识图谱形式的领域约束知识指南，自主评估LLMs的鲁棒性。该方法从领域约束知识图谱三元组系统生成描述性句子以构建对抗性提示，提升评估的相关性与挑战性。这些由LLM自身生成、用于评估其自身鲁棒性的提示，经过严格的过滤与精炼流程，确保仅使用文本流畅度和语义保真度高的提示。这种自我评估机制使LLM无需外部基准即可评估自身鲁棒性。我们通过对ChatGPT等专有模型及Llama-3.1、Phi-3、Mistral等开源模型的广泛测试，评估了框架的有效性。结果证实，该方法不仅减少了对传统数据的依赖，还为受限领域的LLM鲁棒性评估提供了针对性强且高效的手段。
英文摘要：
Traditional methods for evaluating the robustness of large language models (LLMs) often rely on standardized benchmarks, which can escalate costs and limit evaluations across varied domains. This paper introduces a novel framework designed to autonomously evaluate the robustness of LLMs by incorporating refined adversarial prompts and domain-constrained knowledge guidelines in the form of knowledge graphs. Our method systematically generates descriptive sentences from domain-constrained knowledge graph triplets to formulate adversarial prompts, enhancing the relevance and challenge of the evaluation. These prompts, generated by the LLM itself and tailored to evaluate its own robustness, undergo a rigorous filtering and refinement process, ensuring that only those with high textual fluency and semantic fidelity are used. This self-evaluation mechanism allows the LLM to evaluate its robustness without the need for external benchmarks. We assess the effectiveness of our framework through extensive testing on both proprietary models like ChatGPT and open-source models such as Llama-3.1, Phi-3, and Mistral. Results confirm that our approach not only reduces dependency on conventional data but also provides a targeted and efficient means of evaluating LLM robustness in constrained domains.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
