OWL：通过世界文学探究记忆文本的跨语言召回
OWL: Probing Cross-Lingual Recall of Memorized Texts via World Literature
AI总结：
本文通过世界文学数据集OWL探究LLMs的跨语言记忆，发现模型能跨语言召回内容，GPT-4o在新翻译中识别作者和标题达69%。
摘要：
大型语言模型（LLMs）已知会记忆并从其预训练数据中召回英文文本。然而，这种能力在多大程度上泛化到非英语语言或在语言间转移仍不清楚。本文研究了LLMs中的多语言和跨语言记忆，探究当一种语言（如英语）中的记忆内容以翻译形式呈现时，是否能够被召回。为此，我们引入了OWL，一个包含来自十种语言20本书的31.5K对齐摘录的数据集，包括英文原著、官方翻译（越南语、西班牙语、土耳其语），以及六种低资源语言（塞索托语、约鲁巴语、迈蒂利语、马达加斯加语、茨瓦纳语、塔希提语）的新翻译。我们通过三个任务评估了跨模型家族和规模的记忆：（1）直接探测，要求模型识别书籍的标题和作者；（2）名称填空，要求预测被掩码的角色名称；（3）前缀探测，涉及生成续写。我们发现LLMs一致地跨语言召回内容，即使对于预训练数据中没有直接翻译的文本也是如此。例如，GPT-4o在新翻译的摘录中69%的时间识别出作者和标题，6%的时间识别出掩码实体。扰动（如掩码字符、打乱词序）适度降低了直接探测的准确性（打乱官方翻译时下降7%）。我们的结果突显了跨语言记忆的程度，并提供了关于模型间差异的见解。
英文摘要：
Large language models (LLMs) are known to memorize and recall English text from their pretraining data. However, the extent to which this ability generalizes to non-English languages or transfers across languages remains unclear. This paper investigates multilingual and cross-lingual memorization in LLMs, probing if memorized content in one language (e.g., English) can be recalled when presented in translation. To do so, we introduce OWL, a dataset of 31.5K aligned excerpts from 20 books in ten languages, including English originals, official translations (Vietnamese, Spanish, Turkish), and new translations in six low-resource languages (Sesotho, Yoruba, Maithili, Malagasy, Setswana, Tahitian). We evaluate memorization across model families and sizes through three tasks: (1) direct probing, which asks the model to identify a book's title and author; (2) name cloze, which requires predicting masked character names; and (3) prefix probing, which involves generating continuations. We find that LLMs consistently recall content across languages, even for texts without direct translation in pretraining data. GPT-4o, for example, identifies authors and titles 69% of the time and masked entities 6% of the time in newly translated excerpts. Perturbations (e.g., masking characters, shuffling words) modestly reduce direct probing accuracy (7% drop for shuffled official translations). Our results highlight the extent of cross-lingual memorization and provide insights on the differences between the models.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
