PII-TRACE：多轮LLM对话中上下文感知PII检测的基准

中文摘要 AI 辅助

LLM助手和智能体系统会记录长时间的多轮对话。AI提供商通常会在存储或处理对话数据之前，扫描这些对话中的个人身份信息（PII）并进行掩码处理。然而，大多数PII检测器和基准针对的是自包含记录，而非跨轮评估。为了评估多轮对话中跨轮的PII检测，我们引入了PII-TRACE（追踪对话交流中反复出现的PII），据我们所知，这是第一个评估检测器是否能在对话上下文中识别PII，并覆盖跨轮反复出现的标识符的每一次提及的PII基准。PII-TRACE包含13,148个合成多轮对话，涵盖13种语言，带有字符级跨度标注和标识符聚类。在包括前沿LLM在内的十一个基线中，没有检测器能在无PII对话中实现完全的实体级覆盖而不产生大量误报，且单遍阅读在长对话中会丢失三分之一的黄金字符。为缩小这一差距，我们引入了PII-Tracer，一个紧凑的0.6B参数检测器，使用对话级监督进行训练。PII-Tracer在我们评估的所有系统中取得了最高的实体级覆盖率，并在标准单记录基准上也表现强劲。

英文摘要

LLM assistants and agentic systems log long multi-turn conversations. AI providers often scan these conversations for Personally Identifiable Information (PII) and mask the PII before storing or processing conversation data. Yet most PII detectors and benchmarks target self-contained records rather than cross-turn evaluation. To evaluate PII detection across turns in multi-turn conversations, we introduce PII-TRACE (Tracing Recurring PII Across Conversational Exchanges), to our knowledge the first PII benchmark to assess whether detectors identify PII in conversational contexts and cover every mention of a recurring identifier across turns. PII-TRACE contains 13,148 synthetic multi-turn dialogues in 13 languages with character-level spans and identifier clusters. Across eleven baselines, including frontier LLMs, no detector achieves full entity-level coverage without substantial false positives on PII-free conversations, and single-pass reading loses a third of the gold characters on long dialogues. To close this gap, we introduce PII-Tracer, a compact 0.6B-parameter detector trained with conversation-level supervision. PII-Tracer attains the highest entity-level coverage of any system we evaluate and also performs strongly on standard single-record benchmarks.

发表机构

• Perplexity
• Purdue University（普渡大学）
• Rutgers University（罗格斯大学）

机构由 AI 辅助整理，请以论文原文为准。