通过小说中的规范仿像强化大语言模型的隐私推理
Reinforcing privacy reasoning in LLMs via normative simulacra from fiction
AI总结：
本文通过从小说中提取规范仿像，结合监督学习和GRPO强化学习，提升大语言模型的隐私推理能力，验证了小说来源的规范仿像在现实领域中的有效性。
摘要：
大语言模型代理的信息处理实践普遍与用户的情境隐私期望不一致。情境完整性（CI）提供了一个原则性框架，将隐私定义为在情境相对规范下的信息流动。然而，现有方法要么通过监督者-助手架构增加双重推理成本，要么在狭窄任务特定数据上微调。我们提出从小说中提取规范仿像（规范和信息流动的结构表示），并利用监督学习和GRPO强化学习对LLM进行微调。我们的复合奖励函数结合了程序性信号，包括任务清晰度（涵盖模式有效性、构念区分和提取信心）、结构完整性、内部一致性以及情境识别，并引入一个LLM法官评估模型的隐私推理是否基于所持文本的规范宇宙。为缓解过拟合，我们引入了每完成项对比评分：每个完成项都与正确的规范宇宙和随机选择的错误宇宙进行评估，使模型学会根据情境而非记忆源特定规范进行判断。我们在五个对齐CI的基准上进行了评估，涵盖了不同的社会情境，并检验了RL和规范基础的贡献。在七种模型上，SFT引入了保守的先验，限制信息流动，提高了对隐私相关情境的认识，但并未提升隐私判断的正确性。具有规范基础的GRPO在法律合规基准上得分最高，并与众包人类隐私期望有最强的相关性，证明了小说来源的规范仿像可以教给大语言模型情境隐私推理，这种推理能够转移到现实领域。
英文摘要：
Information handling practices of LLM agents are broadly misaligned with the contextual privacy expectations of their users. Contextual Integrity (CI) provides a principled framework, defining privacy as the appropriate flow of information within context-relative norms. However, existing approaches either double inference cost via supervisor-assistant architectures, or fine-tune on narrow task-specific data. We propose extracting normative simulacra (structured representations of norms and information flows) from fiction novels and using them to fine-tune LLMs via supervised learning followed by GRPO reinforcement learning. Our composite reward function combines programmatic signals, including task clarity (subsuming schema validity, construct discrimination, and extraction confidence), structural completeness, internal consistency, and context identification, with an LLM judge that evaluates whether the model's privacy reasoning is grounded in the held-out normative universe of the source text. To mitigate overfitting, we introduce per-completion contrastive scoring: each completion is evaluated against both the correct normative universe and a randomly selected wrong one, teaching the model to condition on context rather than memorize source-specific norms. We evaluate on five CI-aligned benchmarks spanning distinct societal contexts and ablate the contributions of RL and normative grounding. Across seven models, SFT introduces a conservative prior toward restricting information flow, improving recognition of privacy-relevant situations but not the correctness of privacy judgments. GRPO with normative grounding achieves the highest score on a law compliance benchmark and strongest correlation with crowdsourced human privacy expectations, demonstrating that fiction-derived normative simulacra can teach contextual privacy reasoning that transfers to real-world domains.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
