LLM4Trust：探索大型语言模型在信任评估中的能力
LLM4Trust: Exploring the Capabilities of Large Language Models for Trust Evaluation
浏览论文内容
中文总结 AI 辅助
本文提出LLM4Trust基准框架，系统探索LLMs在信任评估中的能力，通过构建信任图、设计属性理解任务及多提示方法，验证其在有限监督下的有效性，并提出防御与批量推理策略。
中文摘要 AI 辅助
信任评估在网络安全中扮演着关键角色，通过支持风险缓解和决策制定来发挥作用。已有多种信任评估方法被提出，其中基于学习的方法提供了高准确性和自动化。然而，这些方法通常需要大量的真实标注数据，训练效率低下，缺乏对基本信任属性的支持，并且可解释性有限。大型语言模型（LLMs）凭借其强大的零样本/少样本推理能力和广泛的知识，提供了一种引人注目的替代方案。为此，我们提出了LLM4Trust，这是第一个系统地探索LLMs在信任评估中能力的基准框架。我们首先构建多样化的信任图来建模五种基本信任属性，并设计相应的属性理解任务。然后，我们评估了八个代表性LLMs在九种提示方法下理解这些属性的能力。基于这一探索，我们确定了最有效的LLM-提示组合，并将其应用于五个真实世界数据集，以验证LLMs的信任评估能力。在此过程中，我们提出了两种从大规模信任图中提取关键信息的策略，以解决LLMs的上下文窗口限制。大量实验表明，LLMs能够有效理解基本信任属性，并在现实世界的信任评估中展现出巨大潜力，尤其是在监督有限的情况下。然而，它们仍然容易受到针对信任图和少样本提示中示范示例的攻击，并且推理成本较高。因此，我们提出了一种防御机制和批量推理，以提高基于LLM的信任评估的鲁棒性和效率。LLM4Trust的源代码可在以下https URL获取。
英文摘要
Trust evaluation plays a critical role in cybersecurity by supporting risk mitigation and decision-making. A variety of trust evaluation methods have been proposed, with learning-based approaches offering high accuracy and automation. However, they often require substantial ground truth, suffer from low training efficiency, lack support for basic trust properties, and provide limited explainability. Large Language Models (LLMs) offer a compelling alternative due to their strong zero-/few-shot reasoning abilities and broad knowledge. To this end, we propose LLM4Trust, the first benchmark framework that systematically explores the capabilities of LLMs for trust evaluation. We first construct diverse trust graphs to model five basic trust properties and design corresponding property understanding tasks. We then assess the ability of eight representative LLMs to understand these properties under nine prompt methods. Based on this exploration, we identify the most effective LLM-prompt combinations and apply them to five real-world datasets for validating LLMs' trust evaluation capability. During this process, we propose two strategies to extract key information from large-scale trust graphs, addressing the context window limitations of LLMs. Extensive experiments show that LLMs can effectively understand basic trust properties and have great potential for real-world trust evaluation, particularly under limited supervision. However, they remain vulnerable to attacks targeting trust graphs and demonstration examples used in few-shot prompting, and incur high inference costs. Accordingly, we propose a defense mechanism and batch inference to improve the robustness and efficiency of LLM-based trust evaluation. The source code of LLM4Trust is available at https://github.com/Jieerbobo/LLM4Trust
发表机构
- Xidian University（西安电子科技大学）
- Purdue University（普渡大学）
机构由 AI 辅助整理，请以论文原文为准。
