发表机构
University of Southern California; Vanderbilt University（南加州大学; 范德堡大学）
机构由 AI 辅助整理，请以论文原文为准。AI 中文总结
针对语言模型均匀词元加权导致记忆表面文本的问题，提出TF-IDF加权交叉熵损失，在保持性能的同时显著减少记忆子串长度，且计算开销低。
AI 中文摘要
大型语言模型通常在均匀词元加权下进行训练，这使得频繁出现且信息量低的词元主导学习过程，并可能增加模型记忆表面文本片段的倾向。为解决这一问题，我们提出了一种信息加权交叉熵损失，利用TF-IDF统计量重新调整词元级别的贡献，强调语义信息丰富的词元，同时降低普遍词元的权重。在五个参数量从1.1B到13B的仅解码器大语言模型上进行的实验表明，该方法在保持困惑度和下游任务性能的同时，持续减少了记忆的子串长度。在LoRA微调下，TF-IDF使所有五个模型的平均子串记忆长度减少了14%。在TinyLLaMA 1.1B上进行全权重微调时，减少幅度达到58%。我们的方法具有架构无关性，可以以不到3%的计算开销集成到现有训练流程中，提供了一种轻量且原则性的方式来缓解记忆问题，而不会干扰标准训练动态。
英文摘要
Large language models are typically trained under uniform token weighting, which allows frequent and low-information tokens to dominate learning and can increase the tendency to memorize surface-level text spans. To address this, we present an information-weighted cross-entropy loss that rescales token-level contributions using TF-IDF statistics, emphasizing semantically informative tokens while down-weighting ubiquitous ones. Experiments on five decoder-only LLMs ranging from 1.1B to 13B parameters show consistent reductions in memorized substring length while preserving perplexity and downstream task performance. Under LoRA fine-tuning, TF-IDF reduces average substring memorization length by 14% across all five models. Under full-weight fine-tuning on TinyLLaMA 1.1B, the reduction reaches 58%. Our approach is architecture-agnostic and can be incorporated into existing training pipelines with less than 3% computational overhead, offering a lightweight and principled way to mitigate memorization without disrupting standard training dynamics.
