arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2610.21333cs.CLcs.CR
收藏

QuasiLeak-Bench:评估基于大语言模型的隐私检测器在策略条件下准证据组恢复任务中的表现

QuasiLeak-Bench: Evaluating LLM-Based Privacy Detectors on Policy-Conditioned Quasi-Evidence Group Recovery

School of Computing, National University of Singapore(新加坡国立大学计算机学院);Agency for Science, Technology and Research (A*STAR), Singapore(新加坡科技研究局)

Yufan Zhu, Yangfan Jiang, Yang Fei, Shizheng Hou, Shubhankar Mohapatra, Xiaokui Xiao

AI总结:

研究针对现有隐私基准未覆盖准证据组推断的问题,提出QuasiLeak-Bench基准,发现现有隐私检测器表现差,微调的4B模型表现优于最强提示式模型,证明任务可学习。

摘要:

自然语言消息常包含准标识符,这些标识符单独使用时作用微弱,但组合起来可共同揭示个人的隐私属性,我们将这种能共同揭示信息的线索集合称为准证据组。现有的隐私基准主要关注显式标识符、属性推断或任务驱动的信息披露,基本未涉及准证据组的推断问题。实际中,披露此类准证据组是否违反隐私还取决于发送方指定的披露策略。我们提出了QuasiLeak-Bench,这是一个用于策略条件下准证据组恢复的基准,用于评估检测器能否恢复完整的准证据组,并标记出给定策略禁止的披露内容。为大规模获取可靠的真实值,我们构建了反向生成流程:首先选择弱准标识符,将其组合成经确认可共同揭示目标属性的准证据组,围绕该证据生成相关文档,再进行自动质量控制。实验表明,现有的隐私检测器在QuasiLeak-Bench上的迁移表现较差,可本地部署的基于大语言模型的检测器远落后于专有模型,主要原因是它们大多完全遗漏了准证据组。不过,在我们的训练集上微调的4B模型,在未见过的文档上的表现超过了最强的提示式模型,这表明该任务是可学习的。

英文摘要:

Natural language messages often contain quasi-identifiers that are weak in isolation but can jointly reveal an individual's private attributes. We call such a jointly revealing set of cues a quasi-evidence group. Existing privacy benchmarks focus on explicit identifiers, attribute inference, or task-driven disclosures, leaving inference from quasi-evidence groups largely unmeasured. In practice, whether exposing such a quasi-evidence group violates privacy also depends on sender-specified disclosure policies. We introduce QuasiLeak-Bench, a benchmark for policy-conditioned quasi-evidence group recovery, which evaluates whether detectors recover complete quasi-evidence groups and flag the disclosures the given policy prohibits. To obtain reliable ground truth at scale, a reverse-construction pipeline first selects weak quasi-identifiers, composes them into quasi-evidence groups verified to jointly reveal a target attribute, and generates documents around this evidence, followed by automated quality control. Experiments show that existing privacy detectors transfer poorly to QuasiLeak-Bench, and locally deployable LLM-based detectors fall far behind proprietary models, mainly because they miss most quasi-evidence groups altogether. However, a 4B model fine-tuned on our training split surpasses the strongest prompted model on held-out documents, showing that the task is learnable.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑