比较预训练与全微调大语言模型嵌入在智人剪接位点基因组数据上的重建攻击
Comparing Reconstruction Attacks on Pretrained Versus Full Fine-tuned Large Language Model Embeddings on Homo Sapiens Splice Sites Genomic Data
AI总结:
本研究针对基因组序列的大语言模型嵌入重建攻击,对比预训练与微调模型的隐私脆弱性,发现微调增强XLNet、GPT-2、BERT等架构的攻击抵抗力,为基因组数据隐私保护提供新视角。
摘要:
本研究调查了应用于基因组序列的大语言模型(LLMs)中的嵌入重建攻击,特别关注微调如何影响对这些攻击的脆弱性。基于Pan等人证明预训练语言模型嵌入可泄露敏感信息的开创性工作,我们使用HS3D基因组数据集进行全面分析,以确定任务特定优化是否增强或削弱隐私保护。我们的研究从三个重要维度扩展了Pan等人的工作:首先,将其重建攻击管道应用于预训练和微调模型嵌入,解决了其方法中未指定嵌入类型的关键空白;其次,实施专为DNA序列定制的专用分词机制,增强模型处理基因组数据的能力,因为这些模型是在自然语言而非DNA上预训练的;第三,对预训练和微调嵌入之间的位置特异性、核苷酸类型和隐私变化进行详细比较分析。我们评估不同类型和维度的嵌入脆弱性,深入了解任务适应如何改变整个基因组序列的隐私风险。研究结果显示预训练和微调嵌入的重建脆弱性存在明显差异:值得注意的是,微调增强了多种架构对重建攻击的抵抗力——XLNet(+19.8%)、GPT-2(+9.8%)和BERT(+7.8%),表明任务特定优化是一种潜在的隐私增强机制。这些结果强调了处理敏感基因组数据的语言模型需要先进保护机制,同时突出微调是值得进一步探索的潜在隐私增强技术。
英文摘要:
This study investigates embedding reconstruction attacks in large language models (LLMs) applied to genomic sequences, with a specific focus on how fine-tuning affects vulnerability to these attacks. Building upon Pan et al.'s seminal work demonstrating that embeddings from pretrained language models can leak sensitive information, we conduct a comprehensive analysis using the HS3D genomic dataset to determine whether task-specific optimization strengthens or weakens privacy protections. Our research extends Pan et al.'s work in three significant dimensions. First, we apply their reconstruction attack pipeline to pretrained and fine-tuned model embeddings, addressing a critical gap in their methodology that did not specify embedding types. Second, we implement specialized tokenization mechanisms tailored specifically for DNA sequences, enhancing the model's ability to process genomic data, as these models are pretrained on natural language and not DNA. Third, we perform a detailed comparative analysis examining position-specific, nucleotide-type, and privacy changes between pretrained and fine-tuned embeddings. We assess embeddings vulnerabilities across different types and dimensions, providing deeper insights into how task adaptation shifts privacy risks throughout genomic sequences. Our findings show a clear distinction in reconstruction vulnerability between pretrained and fine-tuned embeddings. Notably, fine-tuning strengthens resistance to reconstruction attacks in multiple architectures -- XLNet (+19.8\%), GPT-2 (+9.8\%), and BERT (+7.8\%) -- pointing to task-specific optimization as a potential privacy enhancement mechanism. These results highlight the need for advanced protective mechanisms for language models processing sensitive genomic data, while highlighting fine-tuning as a potential privacy-enhancing technique worth further exploration.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。