分词器生育率与基础模型在乌克兰法律文本上的零样本性能：一项比较研究
Tokenizer Fertility and Zero-Shot Performance of Foundation Models on Ukrainian Legal Text: A Comparative Study
AI总结：
本研究比较了七种基础模型在乌克兰法律文本上的分词器生育率和零样本性能，发现分词器生育率差异达1.6倍，Qwen 3模型比Llama系列多消耗60%的token，而NVIDIA Nemotron Super 3 (120B)以更低的成本取得最佳性能，同时揭示了少样本提示在形态丰富语言上的退化以及战时法律语言对模型泛化的影响。
摘要：
在乌克兰法律文本上，不同基础模型的分词器生育率差异达1.6倍，然而这一成本关键维度在模型选择实践中被忽视。我们使用来自乌克兰国家登记册（EDRSR）的273份经过验证的法院判决，对来自五个提供商的七个模型进行了基准测试，测量了分词器生育率以及在三个任务上的零样本性能。发现了四个结果。（1）Qwen 3模型在相同输入上比Llama系列模型多消耗60%的token，使得分词器分析成为成本高效部署的前提。（2）NVIDIA Nemotron Super 3 (120B)取得了最高综合得分（83.1），以三分之一的API成本超越了Mistral Large 3（总参数多5.6倍）——模型规模并不能很好地代表领域性能。（3）少样本提示使性能下降高达26个百分点；分层和提示敏感性消融实验证实，这是乌克兰语演示的内在问题，而非示例选择的伪影。（4）跨时间泛化实验表明，在战前法院判决（2008-2013）上训练的分类器，应用于全面入侵时期的判决（2022-2026）时，性能下降27.9个百分点，并呈现出显著的前后不对称性：较新的模型向后迁移效果更好（比向前迁移高14.6个百分点），但较旧的模型在战时法律语言上完全失败。对于从业者：分词器分析应优先于模型选择，对于形态丰富的语言，零样本比少样本更可靠。为了支持可重复性并解决乌克兰语在法律NLP基准中的缺失，我们发布了一个包含14,452份法院判决的公开数据集，时间跨度为2008-2026年，标注了三个时间段的七个结果标签，这些时间段捕捉了武装冲突对司法程序的影响。
英文摘要：
Tokenizer fertility varies 1.6x across foundation models on Ukrainian legal text, yet this cost-critical dimension is absent from model selection practice. We benchmark seven models from five providers on 273 validated court decisions from Ukraine's state registry (EDRSR), measuring tokenizer fertility and zero-shot performance on three tasks. Four findings emerge. (1) Qwen 3 models consume 60% more tokens than Llama-family models on identical input, making tokenizer analysis a prerequisite for cost-efficient deployment. (2) NVIDIA Nemotron Super 3 (120B) achieves the highest composite score (83.1), outperforming Mistral Large 3 (5.6x more total parameters) at one-third the API cost model scale is a poor proxy for domain performance. (3) Few-shot prompting degrades performance by up to 26 percentage points; stratified and prompt-sensitivity ablations confirm this is intrinsic to Ukrainian-language demonstrations, not an artifact of example selection. (4) A cross-temporal generalization experiment reveals that classifiers trained on pre-war court ecisions (2008-2013) lose 27.9 percentage points when applied to full-scale invasion era decisions (2022-2026), with a pronounced forward-backward asymmetry: newer models transfer backward (+14.6 pp above forward transfer), but older models fail catastrophically on wartime legal language. For practitioners: tokenizer analysis should precede model selection, and zero-shot is a more reliable default than few-shot for morphologically rich languages. To support reproducibility and address the absence of Ukrainian from legal NLP benchmarks, we release a public dataset of 14,452 court decisions spanning 2008-2026, annotated with seven outcome labels across three temporal epochs that capture the impact of armed conflict on judicial proceedings.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
