以人为中心的LLM隐私审计:发现与摩擦
Human-Centred LLM Privacy Audits: Findings and Frictions
AI总结:
本文研究了LLM隐私审计中的人机交互问题,通过LMP2工具发现模型对个人信息的关联预测准确率高,但用户对隐私保护的需求与模型输出的不确定性存在矛盾,提出了九项摩擦并提出未来研究方向。
摘要:
大型语言模型(LLMs)通过从大量训练语料库和用户交互中学习统计关联,并部署系统可以揭示或推断关于个人的信息。然而,人们缺乏实际方法来检查模型关联到他们姓名的内容。我们报告了正在进行研究的中期发现,并介绍了LMP2,一种基于浏览器的自我审计工具。在两项用户研究(总样本量为458)中,GPT-4o对50个日常人物特征的预测准确率高达60%以上,参与者表示希望对LLM生成的关联拥有控制权,尽管他们并未考虑所有输出都涉及隐私问题。为了验证我们的探测方法,我们在公众人物和虚构姓名上评估了八种LLM,观察到稳定姓名条件关联与模型默认值之间存在明显分离。我们的发现还揭示了生成式AI评估危机的更广泛问题:当输出具有概率性、依赖上下文且通过引导用户进行时,模型与个体的关联甚至包括什么仍未明确界定,操作化依赖于设计探测和指标,这些指标难以验证或比较。为朝着可靠、可操作的人为中心的LLM隐私审计迈进,我们识别了在本研究中出现的九种摩擦,并为未来工作和人为中心的LLM隐私审计设计提供建议。
英文摘要:
Large language models (LLMs) learn statistical associations from massive training corpora and user interactions, and deployed systems can surface or infer information about individuals. Yet people lack practical ways to inspect what a model associates with their name. We report interim findings from an ongoing study and introduce LMP2, a browser-based self-audit tool. In two user studies ($N_{total}{=}458$), GPT-4o predicts 11 of 50 features for everyday people with $\ge$60\% accuracy, and participants report wanting control over LLM-generated associations despite not considering all outputs privacy violations. To validate our probing method, we evaluate eight LLMs on public figures and non-existent names, observing clear separation between stable name-conditioned associations and model defaults. Our findings also contribute to exposing a broader generative AI evaluation crisis: when outputs are probabilistic, context-dependent, and user-mediated through elicitation, what model--individual associations even include is under-specified and operationalisation relies on crafting probes and metrics that are hard to validate or compare. To move towards reliable, actionable human-centred LLM privacy audits, we identify nine frictions that emerged in our study and offer recommendations for future work and the design of human-centred LLM privacy audits.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。