PII-Bench：评估查询感知隐私保护系统
PII-Bench: Evaluating Query-Aware Privacy Protection Systems
AI总结：
PII-Bench通过评估查询感知隐私保护系统，揭示当前模型在识别PII相关性方面的不足，提出改进方向。
摘要：
大型语言模型（LLMs）的广泛应用引发了关于用户提示中个人身份信息（PII）暴露的隐私担忧。为了解决这一挑战，我们提出了一种与查询无关的PII遮蔽策略，并引入PII-Bench，这是首个全面评估框架，用于评估隐私保护系统。PII-Bench包含2,842个测试样本，涵盖55个细粒度的PII类别，涵盖从单主体描述到复杂多主体互动的多种场景。每个样本都精心设计，包含用户查询、上下文描述和标准答案，指示与查询相关的PII。我们的实证评估表明，尽管当前模型在基本PII检测方面表现良好，但在确定PII查询相关性方面显示出显著局限性。即使是最先进的LLMs在处理复杂多主体场景时也存在困难，这表明在实现智能PII遮蔽方面仍有大量改进空间。
英文摘要：
The widespread adoption of Large Language Models (LLMs) has raised significant privacy concerns regarding the exposure of personally identifiable information (PII) in user prompts. To address this challenge, we propose a query-unrelated PII masking strategy and introduce PII-Bench, the first comprehensive evaluation framework for assessing privacy protection systems. PII-Bench comprises 2,842 test samples across 55 fine-grained PII categories, featuring diverse scenarios from single-subject descriptions to complex multi-party interactions. Each sample is carefully crafted with a user query, context description, and standard answer indicating query-relevant PII. Our empirical evaluation reveals that while current models perform adequately in basic PII detection, they show significant limitations in determining PII query relevance. Even state-of-the-art LLMs struggle with this task, particularly in handling complex multi-subject scenarios, indicating substantial room for improvement in achieving intelligent PII masking.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
