Lüshi-1840：构建并探究古典中文语言模型的知识截止点
Lüshi-1840: Building and Probing a Knowledge Cutoff for a Classical Chinese Language Model
AI总结：
本文构建了9.83亿参数的古典中文语言模型Lüshi-1840，通过重新标注清洗语料库探究其知识截止点，发现其对1840年后事件的意外值比值显著高于现代通用模型，相关资源将发布。
摘要：
仅在特定年份之前的文本上训练的语言模型可用于研究后见之明与历史语言，但公开的近代中文语料库无出版日期且朝代标签不可靠。本文提出Lüshi-1840，这是一个9.83亿参数的模型，从头开始在12262部书籍（共12.8亿字符）上训练，这些书籍被选定为1840年之前创作的。首个语料库通过通用启发式方法确定日期，实际截止于1644年，但包含20世纪的材料。本文描述了替换该语料库的重新标注与清洗流程，以及使用该流程未采用的标记进行的审计，审计仍发现了后期材料。在模型方面，我们相对于现代参考模型，测量了截止日期后编纂的编年史的意外值：对于1840年后的事件与1820年前的事件相比，该比值为1.38（95%置信区间1.33-1.44），而现代通用模型的该比值约为1.00。在一系列2.94亿参数的模型中，添加后期宫廷记录使该比值从1.33降至1.16，且两个模型在1820年代（添加的宫廷记录开始的时期）出现分歧。该探究追踪了源材料的覆盖范围，未确定单一截止年份；我们的所有探究均未发现经审计的污染。模型、书籍列表及代码将发布。
英文摘要：
Language models trained only on text from before a given year are useful for studying hindsight and historical language, but the open collections of pre-modern Chinese carry no publication dates and unreliable dynasty labels. We present Lüshi-1840, a 983M-parameter model trained from scratch on 12,262 books (1.28B characters) selected as composed before 1840. A first corpus, dated by a common heuristic, in effect ended at 1644 while containing twentieth-century material. We describe the re-dating and cleaning pipeline that replaced it and an audit with markers the pipeline did not use, which still finds later material. On the model side we measure surprisal, relative to a modern reference, on annals compiled after the cutoff. For events after 1840 against events up to 1820 the ratio is 1.38 (95% interval 1.33-1.44), against about 1.00 for modern general-purpose models. In a series of 294M models, adding the later court records lowers the ratio from 1.33 to 1.16, and the two models diverge from the 1820s, where the added records begin. The probe tracks the coverage of sources and does not establish a single cutoff year; none of our probes exposes the audited contamination. Model, book list and scripts will be released.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
