arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2603.24917cs.CLcs.LG
收藏

通过解码约束束搜索估计语言模型中的近原文提取风险

Estimating near-verbatim extraction risk in language models with decoding-constrained beam search

AVERI;Stanford(斯坦福大学);Cornell(康奈尔大学);Google DeepMind(谷歌DeepMind)

A. Feder Cooper, Mark A. Lemley, Christopher De Sa, Lea Duesterwald, Allison Casasola, Jamie Hayes, Katherine Lee, Daniel E. Ho, Percy Liang

AI总结:

本文提出解码约束束搜索方法,以有效估计语言模型中的近原文提取风险,揭示更多可提取序列及模型规模对风险的影响。

摘要:

近期研究表明,标准贪心解码提取方法在量化LLM中的记忆程度时,未能捕捉到不同序列中提取风险的变化。概率提取——在解码方案下计算给定前缀生成目标后缀的概率——解决了这一问题,但仅适用于原文记忆,无法处理类似隐私和版权风险的近原文实例。量化近原文提取风险成本高昂:近原文后缀集是组合爆炸级的,可靠的蒙特卡洛估计可能需要每个序列约100,000个样本。为缓解这一成本,我们引入了解码约束束搜索,该方法在成本与约20个蒙特卡洛样本相当的情况下,能提供近原文提取风险的确定性下界。在实验中,我们的方法揭示了原文方法无法察觉的信息:更多可提取序列、每序列显著更大的提取量,以及近原文提取风险在模型规模和文本类型上的表现模式。

英文摘要:

Recent work shows that standard greedy-decoding extraction methods for quantifying memorization in LLMs miss how extraction risk varies across sequences. Probabilistic extraction -- computing the probability of generating a target suffix given a prefix under a decoding scheme -- addresses this, but is tractable only for verbatim memorization, missing near-verbatim instances that pose similar privacy and copyright risks. Quantifying near-verbatim extraction risk is expensive: the set of near-verbatim suffixes is combinatorially large, and reliable Monte Carlo (MC) estimation can require ~100,000 samples per sequence. To mitigate this cost, we introduce decoding-constrained beam search, which yields deterministic lower bounds on near-verbatim extraction risk at a cost comparable to ~20 MC samples per sequence. Across experiments, our approach surfaces information invisible to verbatim methods: many more extractable sequences, substantially larger per-sequence extraction mass, and patterns in how near-verbatim extraction risk manifests across model sizes and types of text.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑