RedactionBench：基于上下文完整性的隐私保护基准测试
RedactionBench
AI总结：
RedactionBench通过200个跨11个领域的文档，评估红actions的上下文隐私问题，提出R-Score指标，揭示红actions的主观性，推动隐私保护系统的发展。
摘要：
大型语言模型日益应用于需要擦除个人身份信息（PII）的敏感领域。尽管擦除PII是数据清理的必要步骤，现有基准测试将提取机制与隐私语义混为一谈。公共电话号码与医疗记录中的电话号码并不等同。是否构成违规取决于持有者、原因和上下文，根本区别红actions与简单实体识别。基于上下文完整性，我们引入RedactionBench，一个手动标注的基准测试，包含200个跨11个领域的文档，主要源自真实世界来源。我们还引入R-Score，一种新的字符级指标，将语义相似的红actions视为同等重要，并消除浅层格式选择，如电话号码的不同遮蔽样式。在命名实体识别模型、实体提取小型语言模型和前沿模型上进行评估，证明上下文红actions仍是一个未解决的问题。对RedactionBench的80多名用户的人工评估显示隐私观念存在明显分歧。标注者在强制性红actions（89.4%）和安全文本保留（94.1%）上达成一致，但在上下文红actions（47.7%）上未能达成一致。这种差异展示了上下文隐私的主观性，推动R-Score，将上下文模糊性与严格精度分离。我们比较了35个模型家族的性能，并报告了它们在擦除PII方面的表现。最后，我们发布RedactionBench，以建立未来隐私保护系统的基准，希望激发高效模型设计和标准化评估。
英文摘要：
Large Language Models are increasingly applied to sensitive domains that require redaction of personally identifiable information (PII). While redacting PII is a data cleaning prerequisite, existing benchmarks conflate extraction mechanics with privacy semantics. A public phone number is not equivalent to a phone number in a medical record. Whether information constitutes a violation depends heavily on who holds it, why, and in what context, fundamentally differentiating redaction from simple entity recognition. Grounded in contextual integrity, we introduce RedactionBench, a manually annotated benchmark comprising 200 diverse documents across 11 domains, mostly seeded from real-world sources. We also introduce R-Score, a novel character-level metric that treats semantically similar redactions equally and nullifies shallow formatting choices, such as varying masking styles for phone numbers. Evaluations across Named Entity Recognition models, entity extraction Small Language Models, and frontier models equipped with agentic tools demonstrate that contextual redaction remains an unsolved problem. A human evaluation with over 80 users on RedactionBench reveals a stark dichotomy in privacy perceptions. Annotators show consensus with target labels for mandatory redactions (89.4 percent) and safe text preservations (94.1 percent), but fail to agree on contextual redactions (47.7 percent). This variance demonstrates the subjective nature of contextual privacy and motivates R-Score, which decouples contextual ambiguity from strict precision. We compare 35 models across families and report their performance in redacting PII. Finally, we release RedactionBench to establish a baseline for future privacy-preserving systems, hoping to inspire efficient model design and standardized evaluations.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
